MG-RAST newsletter, September 2016

Contents

  1. Use version 4.0 prototype for reduced wait times for job completion
  2. Recent technical issues
  3. New web user interface taking shape
  4. Workshop announcement
  5. Miscellaneous

—————————–

1. Use version 4.0 prototype for reduced wait times for job completion

Wait times for MG-RAST pipeline completion have been steadily rising for the last year. We are introducing changes to the pipeline and the database technology to accelerate processing and loading. Currently there is a significant backlog, over 4 Terabases. To alleviate this situation, we will introduce a different database technology that speeds up loading dramatically. However we will not be able to port the existing web interface to this new technology.

For a little while, MG-RAST will send out two notifications for each job completed: (1) job is available to view, analyze and compare in the version 4.0 prototype (see below, item 2) and the API. (2) A second email will be send once the loading into the old database technology for version 3 is completed. As the prototype for version 4.0 is nearly 100% complete and has 95% of the feature of version 3.0 we think this solution will provide the shorter round trip times we are looking for. This situation will be changed once we migrate to the new version.

2. Recent technical issues

In addition, as some of you have certainly noticed, MG-RAST was down four times in this quarter. We experienced 2 planned and 2 unplanned power outages that caused significant problems. While the majority of the data storage systems that support the MG-RAST API handled those outages well, our SQL servers (Postgres) did not. This caused a massive amount of downstream work. Unfortunately the 2nd unplanned power loss interrupted the clean-up of the first, creating a total of 4409 data sets that were incorrectly or incompletely loaded into the SQL representation. Currently the SQL databases support the Analysis and Download pages, thus rendering those jobs inaccessible.

We have begun re-loading the affected data sets into the SQL databases. Affected users will be notified once their data sets are loaded correctly.

The SQL databases are a bad match for the volume of data that MG-RAST handles today. We are working to replace the SQL database with a more modern (“sharded”) database in a future release. Unfortunately we cannot make this change while we are still using the old web user interface.

3. New web user interface taking shape

As mentioned in a previous newsletter, the web user interface for MG-RAST is undergoing significant changes. Users commenting on significant wait times and timeouts as well as the extremely high load on our servers prompted us to pick a new approach for web interface design.

The new interface features:

  • a cleaned up fresh look with an extended set of features
  • compatibility with all modern browsers
  • fully API-based with faster response times
  • client side computing for next to instant analysis results
  • provenance information and export functionality on all data

As some of you might recall the existing web user interface pre-dates the MG-RAST API, it accesses the underlying infrastructure directly, making changes next to impossible. Therefore we need to move off the old web interface and onto the modern interface as quickly as possible to increase our throughput and prevent future issues like those discussed in item #1.

A beta version of the new user interface is available at http://v4-web.metagenomics.anl.gov. We appreciate your feedback!

Feedback URL: https://goo.gl/forms/OaNnqBljkXinmglF3

Once the new version is fully tested, we will retire the old web interface within 4-5 weeks.

 

4. Workshop announcement

The MG-RAST team is accepting proposals for MG-RAST workshops for 2017. If you are interested in hosting a workshop and you can accommodate 20-30 students (who are expected to bring their own laptop or make other arrangements) and have decent WIFI and internet connectivity, contact us via the help-desk email that you are familiar with.

We are seeking to run at least 3 workshops each aiming at users at various levels. The 3-day workshop is increasing the amount of computational skills required with each day, starting with novice users and ending (on day 3) with an introduction to the API and other advanced topics.

Day 1: introduction and first steps

  • Gentle introduction to MG-RAST and shotgun metagenomics
  • The MG-RAST web interface
  • Performing analysis with the MG-RAST web user interface

Day 2: web based analytics and first steps on the command line

  • Advanced analytics with the web frontend
  • Using metadata
  • Using R, Perl and Python scripts to perform data uploads and analysis

Day 3: advanced topics

  • More details on using R, Perl and Python scripts to perform data uploads and analysis
  • The MG-RAST API
  • How to change the MG-RAST pipeline and/or set up your own MG-RAST like pipeline (“Skyport, Shock and Awe”)

5. Miscellaneous

  • We will remove all old content from the Wiki with the release of version 4.0

Comments are closed.