MG-RAST newsletter November 2016

Contents

  1. Switching to new MG-RAST web interface  on November 15th 2016
  2. Update on processing backlog
  3. Feedback
  4. Miscellaneous

—————————–

1. Switching to new MG-RAST web interface on November 15th 2016

After an open beta testing period and feedback from a significant number of users, we are now ready to transition away from the old MG-RAST web interface to a new more modern and more capable user interface.

–> http://metagenomics.anl.gov will direct you towards the new style interface after November 15th. <– 

As many of you have tested the new interface already (at http://v4-web.metagenomics.anl.gov) we feel confident that the new version will add significantly to the usability of the system, the old style interface will remain available for a while at (http://v3-web.metagenomics.anl.gov).  The old style interface will not be showing any new data, but remain functional for a while.

2. Update on processing backlog

Due to both the massive demand for MG-RAST analysis and several unplanned power failures, we experienced problems with loading the results of the analysis pipeline (aka profiles) into the database that hosts said profiles. We have now transitioned the pipeline (as of November 9th 2016) to load into Cassandra instead of Postgres and are able to load data at a rate of 1 Gigabasepair / 60 seconds. We already have uploaded 30% of all public data to Cassandra and are continuing to transition all profiles to Cassandra in the coming weeks.

 

Technical details below:

We are working with a 3 step plan that we present here in the interest of transparency:

– step 1:

November 9th: Switch over API and pipeline to the new (v4 or “Cassandra” version) of the code base. Any new jobs created will automatically use the new code. The old style API will remain available for a while and we will provide automatic forwarding in the API. If the automatic forwarding does not work for you, you can reach the old style (“postgres” or v3) API at http://api-pql.metagenomics.anl.gov). You should not need to use the temporary address, but some codes might not contain support for http redirecting.

The following API resources are being forwarded to the v3 API:

/matrix

/annotation

Consequently they will not be available for any new data sets being added to the Cassandra database until we have retargeted them. This work is going on now.

– step 2:

We are currently loading all public data sets into the v4 database and this process will continue for a while (15,000 data sets took 1 week to load).

A large number of data sets are currently being processed. All existing jobs (they are mostly waiting for the database load steps) are being rewritten to load into the v4 databases as well. This process will start November 10th 2016 and continue for a number of days.

– step 3:

On November 15th we will switch the CNAME metagenomics.anl.gov to point to v4-web.metagenomics.anl.gov. (The old style interface will remain available at v3-web.metagenomics.anl.gov).

On or around November 15th we will deploy the code to retarget the /annotation API resource to the v4 database (Cassandra).

The /matrix API resource will be retargeted at a later date. NOTE: Until it is retargeted the /matrix API resource will only support data loaded prior to October 1st 2016.

3. Feedback

We will keep the survey for the new web interface  (located at http://v4-web.metagenomics.anl.gov ) open at https://goo.gl/forms/OaNnqBljkXinmglF3 Please continue to provide us feedback for the changed web interface. The more specific you are, the better we are able to build what you need.

4. Miscellaneous

  • We have updated the manual for version 4.0
  • We will remove all old content from the Wiki with the release of version 4.0

Comments are closed.