The “Shellshock” vulnerability is fixed

Too long; didn't read: We have sent out an automatic update to address this issue. If you have not disabled automatic updates, or blocked it in your firewall your system in updated now. Updates are automatically so you do not have to take any action to get them.

Shellshock is one of the gravest vulnerability to hit Unix like operating systems ever. Everything from CGI scripts to DHCP client is affected. It lays countless websites, servers, PCs, OS X Macs, routers and other infrastructure open to hackers.

Searchdaimon does use Linux and did ship with an effected version of bash, the program that has the vulnerability. However none of the web facing services, like CGI scripts being executed by the web server or the ssh server was affected. However sins Searchdaimon is open source users may have made changes, or installed other software on their machines in addition to Searchdaimon.

There was also the issue with the DHCP client. Most Searchdaimon installations are server, and have a static IP address, but for thus that do use DHCP a malicious DHCP server on the local network could allow an attacker to run arbitrary code on the client machine.

We have therefore more or less been in emergency mode since we got the first warning email about this. However since this effected only a single install package, we made a fix within 15 minutes, sent it out to beta tester within two hours and updated everyone the next day.

The fix is now added to the automatic update system, so if you have not disabled automatic updates, or blocked it in your firewall your system in updated now.

More information about the Shellshock vulnerability in general is available at https://access.redhat.com/node/1200223 .

More information on our fix and how to do a manual install is available in our forum her: Update to address the “Shellshock” vulnerability is being sent out immediately (26.09.2014) .

Searchdaimon Enterprise Search is now free and open source on GitHub

We are proud to announce that the source code for Searchdaimon ES is now open source and freely available on GitHub under the GPL v2 license.

This is an exciting time for us. Together with the open source community we hope we can build one of the best solutions for enterprise search yet.

Searchdaimon ES is a search engine for corporate data and websites. The ES comes with a powerful administrator interface and can index websites and several common enterprise systems like SharePoint, Exchange, SQL databases, Windows file shares etc. The ES supports many data sources (e.g., Word, PDF, and Excel) and the possibility of faceted search, attribute navigation and collection sorting.

The ES is normally deployed as an integrated suite with search software and Linux that one normally run as a virtual machine on cloud providers like Amazon Web Services, on one’s own hardware as a virtual machine using VMware, Xen or VirtualBox or directly on a physical server. The ES is mostly written in C and Perl. The company was founded in 2005.

• Official press release
• GitHub repository

 

More information about the product and screenshots are available at the bottom of this post.

General

• Road map to the future
• When to use and not to use the ES
• Help us translate the user interface to different languages
• How to upgrade existing installations to the open source version
• How Searchdaimon As, our company will make money

 

The release

We have released all of our source code related to the Searchdaimon enterprise search engine to GitHub, with full commit history. We will be using the GitHub repository internally as our only source code repository from now on. It is currently about 100K lines of code. These is the full source code, tools and build chain to build the ES.

The only exception is that we unfortunately have had to hold on to some related source code that uses proprietary code from 3-partys. However none of thus are essential, and binaries are made freely available for most of it. A list of non-released code is available at: Statement on non-open source code.

Try it out

An online demo is available at http://www.searchdaimon.com/demo/ (no registration is required)

Download

• Download prepackaged virtual machine images for VMware, Xen and VirtualBox
• Instructions on how to run on Amazon Web Services
• How to install from an CD or ISO file
• The source code and instructions on how to get started is available at GitHub

Amazon Web Services price decrease

We are excited to announce that both we and Amazon Web Services are lowering our prices. Together this price reduction will make a standard medium instant 21% cheaper to run.

Some examples

	Old price	New price	Decrease
Standard Small (m1.small)	0.120	0.095	21%
Standard Medium (m1.medium)	0.240	0.190	21%
Standard Large (m1.large)	0.480	0.390	19%

Prices include the Searchdaimon ES software and Amazons EC2 fees, but not EBS fees and data transfer fees. For the US EAST (Virginia) region by the hour spot price. Prices for reserved and spot Instances will be lower. The price reduction will take effect immediately.

Amazon Web Services

As you may know Searchdaimon is available at Amazon Web Services (AWS). You can buy an Searchdaimon ES instances by the hour. You have no commitment beyond an hour and can come or go as you please.

If you have predictable, steady-state workloads, you can also reduce your cost further by buying AWS instances for a whole one year or three year.  This is equivalent to buying or leasing servers, like folks have done for years, only it is in the cloud. In both cases you will benefit from this price drop.

You can see a full list of the new, lower, pricing on the Searchdaimon AWS pricing page, and read more about running the Searchdaimon ES on AWS at the Searchdaimon AWS starting guide.

Searchdaimons CTO interviewed by Beyond Search

Beyond Search has an interview with Searchdaimons CTO as part of their Search Wizards Speak series. The interview gives a good introduction to our company, further plans and history.

Link: http://www.arnoldit.com/search-wizards-speak/searchdaimon.html

Searchdaimon ES available in the Amazon cloud; Named Amazon Web Services partner

You can now deploy Searchdaimon ES as a virtual machine in the Amazon cloud with a single click You only pay for the number of hours you are running it. There are no signup fees and no hard limits on document numbers, numbers of users or the number of data sources one can index. Support is also included.

Getting started guide: http://www.searchdaimon.com/aws/
More info about Searchdaimon ES at Amazon:  https://aws.amazon.com/marketplace/pp/B008XME9MM

Named Amazon Web Services partner

Searchdaimon has also been named an Amazon Web Services (AWS) partner, and is among the first AWS Partners ever to gain public designation as a Standard Technology Partner. This will give Searchdaimon greater access to the technical team at AWS, ensuring that our search technology continues to meet your needs.

Follow us on Twitter and Google+

Searchdaimon is now on both Twitter and Google Plus :) 

( And have been on Facbook for ages )

Several new features in next Searchdaimon ES version

We are in the final stages of rolling out the next version of Searchdaimon ES. It has several new features, including:

Administration panel

List documents in a collection

You may now list the documents in a collection.

 

Document information

You can also view info about each document directly from the administration panel. Great for viewing the result of data conversion, attributes and access control info.

 

Crawler console

Crawlers will output information on what they are doing, in real time.

 

Search

Some changes to the ranking algorithm to boost documents that has hits in the title or file name.

Core functions

• Support for setting your own crawler delay (delay in seconds between successive crawl requests to the same data source).
• The first time wizard is gone.
• Support for not having any end user management system ( Active Directory etc ).
• Attribute search also for public collections.
• Support for query caching.
• ++ several bug fixes.

 

This version will be automatically sent out to everyone during August.

One click launch of ES on Amazon Web Services

You can now deploy Searchdaimon ES on Amazon Web Services with a single click! Just go to https://aws.amazon.com/amis/searchdaimon-enterprise-search-2-5 and use the Launch AMI button, or read the how to on running ES on Amazon ec2.

 

You can also make your own custom Amazon machine image if you have very special needs, please see: Create your own Amazon machine image.

Searchdaimon CTO speaking at Enterprise Search Europe 2012

Our CTO Runar Buvik will be giving a speech about how to evaluate different search technologies at the Enterprise Search Europe 2012 conference.

More info is available at http://www.enterprisesearcheurope.com/2012/Programme.aspx#session_5474

 

If anyone is interested in getting together during the conference, please feel free to email him at: rb [at] searchdaimon [dot] com .

Searchdaimon.com gets an update

We have made a major overhaul of the searchdaimon.com website. Enjoy

Before and now:

Some of the source code for crawling has also been made public available at http://www.searchdaimon.com/source/

Launching a free enterprise search appliance

We have launched a free version ow our enterprise search appliance, Searchdaimon Enterprise Search. The free version has all the functions of the paid, except it is only licensed for 10 end-users, if used together with a single sign-on system like Microsoft Active director. There is no limitation on usage when used as a public websearch or other systems where your user don't have to login with a separate user account to search.

More information is available in the official press release: Searchdaimon Launches Free "mini-Google" for Companies.

Also check out the demo, or simply download it today ( no registration required ).

New content federation function

Structured data can now be presented as a table among the results. Often you'll see the information you need directly in the user interface, without having to open the data source itself.

Here you see a hit from a SharePoint list of contacts. Notice how the email address is directly visible in the results.

ES accepted into the VMware marketplace

We have been accepted into the VMware Virtual Appliance Marketplace

We are very excited, the VMware marketplace is fast becoming for enterprise application what "app store" is for the iPhone.

The VMware marketplace is collection of more than a 1500 cloud ready virtual applications. A virtual appliance is a virtual machine image designed to run on a virtualization platform. Normal it contains a striped down Linux operating system and application software fully configured, ready to run, without the need for any local installation or configuration. This makes it easy to deploy new applications in the cloud. The performance loss is only ~3-10% compared to installing it on a dedicated server.

Link to our listning: https://solutionexchange.vmware.com/store/products/1572
We currently rank as nr. 1 on the search query enterprise search .

( VMware can in no way be held responsible for any content you may access true the VMware Marketplace )

Integrated with 24sevenoffice, 30 000+  new users using our technology

24sevenoffice have started using Searchdaimon ES to provide search in their webbased application. All users now have access to search their files, email, email attachments, crm notes and project documentation.

Searching is seamlessly integrated with the 24sevenoffice design. User queries are sent to an ES server which returns XML. 24sevenoffice stores their data in MSSQL databases and SANs accessible by SMB. Using the ES crawler extensions, they created a Perl crawler that connects to the appropriate databases and SANs to download new data. The ES also integrates with their custom made permission system, only providing search results with resources a user is authorized to see.

Usage example

In the background you can see the 24sevenoffice ajax user interface. I have searched for "runar buvik".

1. Query box where users ask queries.
2. A Word file in the search results. With filename, filetype icon and an abstract of the document.
3. A filter where users can select to filter out results which isn't from a specific filetype, data range and/or module.

 

Update 20 March 2013. 24SevenOffice has also made a blogpost on this: Beta for free-text search in collaboration with Searchdaimon.

About 24sevenoffice

24sevenoffice is a leading on-demand Software as a Service (SaaS) vendor for integrated business management software. Ther main product is an integrated CRM and ERP system, that also has filestorage and email accounts. 24SevenOffice is a pioneer in the development of Internet based applications and was founded in 1996. http://www.24sevenoffice.com

New Searchdaimon.com website

We have completely redesigned the entire site, and added a new community section. All suggestions and feedbacks are most welcome and much appreciated.

Looking for alpha testers

We will start offering the possibility to install ES on a server you already own. To do this one boot a server using a special CD that will wipe the server clean, then install the ES software. We need your help to test this process on as many servers as possible. To participate you need a spare server with 64 bit possessor and SATA or SCSI disk. Then mail support [at] searchdaimon [dot] com .

ES on physical server [BETA]

We are pleased to announce that you now can install ES on your own physical server. If you wan't to help us test this, contact support for instructions and a free test license.

Ps: Please keep in mind that this is a true beta period. Not a clever promotion tactic, so expects some bugs.

Searchdaimon on Facebook

Come join us on Facebook. As all hip companys Searchdaimon also have a Facbook profile. Become an fan and get the news streamed to your news feed.

Join us at: http://www.facebook.com/pages/Searchdaimon/143499632360166 .

New attributes/meta-information function

Documents, crm-systems and e-mail servers may be a valuable source for meta-data, such as author, source, project etc. This meta-information can be saved with each document and used for filtering. We have had support for meta-information in the core search kernel for some time, and you have probably used it to filter on dates and file types.

In v2.1 meta-information has been extensively expanded, and almost every document now has some kind of metadata. The first thing you'll notice is the more advanced filtering menu on the left of the search interface. The default menu will show document types subdivided into file formats, information type for particular useful meta-data, and date of creation.

On the search results page we now display who emails are to and from. You may click on the email address to filter on it. Clicking on the magnifying glass lists all emails from this email address.

On some documents, selected meta-information will be shown in the snippet, too.

It is also possible to search for specific meta-information. For instance, the query 'attribute:"author"' will return all documents with an author set, and the query 'book attribute:"author=John Doe"' will return all documents of the specified author containing the word 'book'.

 

Meet us at SearchSummit Norway 2008

We are happy to announce that Searchdaimon will be a partner for the SearchSummit Norway conference.

If you want to learn more about our technology and meet us face to face, drop by our exhibition booth. SearchSummit Norway will also include speakers like Susan Feldman (IDC), Chris Mahoney (Second Life) and John Markus Lervik (FAST). All major companies in the search business in Norway are expected to be there.

The conference is in Oslo, on the 3rd of April. Participation fee is 1290 NOK. However, if you contact us soon, we may be able to give you a pass for free.

ExactSeek partners with Searchdaimon

Searchdaimon partnered with Jayde Online to deliver the search technology for their web search engine ExactSeek.com. Using our search platform, ExactSeek's index has grown from 100 million web pages to over 500 million and will increase to 1 billion in the coming months. News articles (in Norwegian) http://www.digi.no/505126/norsk-soketeknologi-til-amerikansk-sokegigant http://www.universitetsavisa.no/ua_lesmer.php?kategori=nyheter&dokid=4798908ae99126.78404924 Full press release:

ExactSeek Partners with Searchdaimon AS on New Search Engine Technology

ExactSeek implements Searchdaimon AS advanced search technology to provide improved search speed, relevancy and index scalability.

ExactSeek.com, a web search engine committed to providing small and medium-sized businesses with affordable search engine advertising alternatives to pay-per-click, today announced an exclusive partnership with Searchdaimon AS, a technology company located in Trondheim, Norway that has developed an advanced search technology capable of high speed indexing and search of huge amounts of unstructured data.

Searchdaimon has long been a participant in the well known search technology environment at the Norwegian University of Science and Technology along with other companies like Google, Yahoo and Fast.

"Searchdaimon's search technology focuses on scalability through parallelization and advanced text analysis for relevancy," said Runar Buvik, CTO at Searchdaimon. "Using our search platform, ExactSeek's index has grown from 100 million web pages to over 500 million and will increase to 1 billion in the coming months."

ExactSeek's implementation of the Searchdaimon search platform features rapid crawling and indexing of submitted URLs, incorporation of visitor input through search result voting, a comprehensive news search, a paid inclusion program and keyword ranking tools. Future plans include the release of a distributed, downloadable crawler that can be employed by any PC user to further expand the ExactSeek database and customized search options for webmasters and site owners interested in providing web search to their visitors with their own website look and feel.

"ExactSeek has been focused on improving its search technology for several years. Our partnership with Searchdaimon has enabled us to address the fundamental search issues of speed, relevancy and scalability," said Mel Strocen, CEO of Jayde Online, Inc., ExactSeek's parent company. "Most importantly, it provides us with the opportunity to further engage the webmaster community in our growth and to provide a variety of search distribution and advertising options to our partner network which includes the ISEDN (Independent Search Engine & Directory Network)."

ExactSeek's beta search is available at http://www.exactseek.com

About ExactSeek:
ExactSeek.com is part of the Jayde Online Network which includes GoArticles.com | SiteProNews.com | DropJack.com | Blog-Search.com and FreeWebSubmission.com. Jayde Online Inc. has been internet-focused from its inception in 1998 and primarily involved in the publication of email newsletters and the development of niche and general search engines. ExactSeek was founded in 2002 and has offices in Winnipeg, Canada and San Jose, California.

For more information visit: http://www.exactseek.com

Contact Information:
Barry Green
ExactSeek.com
Vice President, Business Development
http://www.exactseek.com
408-482-5644

Enterprise search, first release

We are proud to announce that our enterprise search appliance, Searchdaimon ES is general avalibal.

Administration interface

We are nearing us the first release of our enterprise search product, one of the remaining tasks has been to create a administration interface for our users. This is now close to being complete. Here's a sneak preview:

Overview:

This the main view, where it's possible to manage all the collection that have been added.

Managing a collection

The view you get when altering the data for a single collection.

Scan results

This is the results you get after the local network has been scanned for possible collections.

Dagur Johannsson and Eirik Nygaard joins the team

We have hired two new developers. Both are students at NTNU, and will be working part time.

Dagur Johannsson Dagur is currently on his third year studying Bachelor in informatics at NTNU. His main focus in computer science is information management and programming, an activity he has done for 5 years. He has developed several applications toward the end user, and thus finds it important to stress user friendliness.

 

Eirik Nygaard Mr. Nygaard is currently on his second year studying computer science at NTNU. He has over eight years of programming experience, for most of this time he has been working on open source projects related to operating systems kernels, and low level userland applications and libraries.

He is also a developer for DragonFlyBSD, and the head of Programvareverkstedet one of two student computer clubs at NTNU. Read more at http://www.searchdaimon.com/people.htm

We are changing our name to Searchdaimon

Daimon is an alternative spelling for daemon, and a captain rank in Star Trek, see http://en.wikipedia.org/wiki/Daimon

New office

We moved into offices in Gløshaugen Innovation Center, in Trondheim to day. The management has had offices in Oslo Innovation Centre for some time now, but the developers, Runar Buvik, Magnus Galåen and Martin Breda, which lives in Trondheim hasn’t had any place to stay, until now.

For any start up company, getting the first real offices is a large milestone, sow we are very excited. Below are images of our name on the door, Runar and Magnus working, and the view from the windows.

Even more storage servers

The crawl has unfortunately been down for a while now. We managed to run out of space to store new pages. We believed we hade ordered new servers in good time before we needed them, but the ones we wanted were temporally out off stock from our supplier. When we finally did get them, we hade some software problems that we had to addressed before we where able to use them.

Guess the “just in time” strategy isn’t the best for buying hardware after all. It do work now. Sow total storage space is now 4.5 TB. See here for live stats: http://bbh-001.boitho.com/phpSysInfo/

200 million crawled pages!

We have reached 200 million crawled pages. Se live status here: http://dcsetup.boitho.com/cgi-bin/dc/topCrawlers.cgi We are crawling ca 120 million pages a month now, so 300 million isn’t far away.

See status for a page

Now you can get status for your pages.

Just submit your url here: http://bbh-001.boitho.com/cgi-bin/UrlToDocID/index.cgi , like this: http://bbh-001.boitho.com/cgi-bin/UrlToDocID/index.cgi?url=http%3A%2F%2Fwww.boitho.com

New beta index

http://www.boitho.com/beta/

Two new storage servers

Installed two more storage servers today. Bringing total storage space up to almost 3 TB. The servers are Super Micro 5014C-MT. P4 with Hyper-threading, 512 MB ram and 900 GB disk space.

Photo: Supermicro Inc

See here for an overview over our storage system: http://bbh-001.boitho.com/phpSysInfo/

Incorporated!

Got the papers from Brreg (The Norwegian Register of Business Enterprises) today. Boitho AS have been incorporated.

Taken up into Oslo Innovation Centres incubator

We have had offices in Oslo Innovation Centre for some time now, and today we where official taken in into there incubator. http://www.forskningsparken.no/companybase/showcompanypublic.aspx?companyID=8501

Enterprise search

For some time now, we have been looking into using the Boitho technology to create an enterprise search engine. Today it is official; we are creating an enterprise search group to develop solutions for enterprises.

This won’t just be a patch to our existing technology, but a whole new technology in its own tight. But hopefully the web and enterprise systems can be friends, sharing common (open) api’s.

This whey one can plug normal Boitho nodes into the enterprise search system where one want to search large data collections that don’t have document lever access rights and all the other stuff that slows down enterprise search engines.

Distributed crawler downtime

The server used to distribute urls for download are having disk problems. We are useing two SCSI disks in raid1 (mirroring), one of this disk have fallen out. It don't appear to be physical broken, only fallen out of the raid. We will have to rebuild the raid and see if it will start to work again. This is 140 GB disks sow this will take some time, probably 10-12 hours. The crawling will be down for this period.

Boitho discover

Wondering what happens to the data you crawl? Is the project even a live? Why aren't we release a new index? At least now you can brows the data online: http://www.boitho.com/discover/

Search engines against comment spam

Google is announcing the idea of a rel="nofollow" tag on individual links. Forums, blogs, guest books, and others who allow the public to add text and links by themselves, can tell search engines that the links are not necessarily approved by the page and should be treated as untrusted.

These links shouldn't be factored into search engines ranking calculations as a vote from the page they occur on. This means that adding comments as "Visit my discount pharmaceuticals site" or "Buy cheap Rolexes here" with a link to a spam page, to boost their link population and author text becomes less attractive to spammers.

Boitho, as most search engines, will support this tag. Related Announcements: Google Msn search Yahoo Searchenginewatch

Getitng more crawlers

Thanks to Anders Christensen and The Department of Computer and Information Science at The Norwegian University of Science and Technology (NTNU) for lending us three old computers as well as, hosting space and Internet access.

When the servers get operational you can see them crawl as "idi-ntnu" at the user statistics page http://dcsetup.boitho.com/cgi-bin/dc/topCrawlers.cgi.

Hardware problems

One of our storage servers have been behaving strange lately. It could behave normal for 2-3 days, and then crash without any warnings. We couldn't find any thing wrong, and look at it for days. We first believed that it what a bug with the indexing software because it always was the program running when this happened.

It turn out that the cpu fan from time to time would stop, then start again. If we at the same time where using all the disks in the box, by running the indexer, the whole system would get to hot, and the cpu would overheat, crashing the server.