Archive for September, 2014

MG-RAST newsletter, September 2014

Wednesday, September 17th, 2014

************************************************
Request for letter of support/testimonials
************************************************
The MG-RAST team is applying for funding to support continued development on MG-RAST and to improve several aspects of the system. In particular, we need to update and improve the underlying data integration which has not changed significantly in the last five years. The funding will also be used to develop and upgrade the user interface.

We are requesting you as a member of our user community to help us by providing a letter of support that addresses the potential for the project, along with a testimonial as to the value MG-RAST has brought to your research. Your opinion is very important to us, we would like to include testimonials from the entire spectrum of MG-RAST users.

Your letter of support should be addressed to Dr. Folker Meyer at folker at anl.gov and include your name, position and organization, it will need to be received before September 30, 2014 for inclusion in our proposal.

Click here to start

Thank you for your help,
Folker Meyer and the MG-RAST team

************************************************
Recent highlights
************************************************

– More server stability (new backend technology)

You may or may not have noticed several hardware related outages recently. Several of our servers are getting rather old and instead of replacing in kind (which is hard at our budget levels) we chose to move to newer more flexible technologies. Over the past 12 months, we have re-written the entire MG-RAST storage subsystem to rely on an object management system [SHOCK (https://github.com/MG-RAST/Shock)] rather than a traditional file system.

Used together with the AWE resource management software, SHOCK allows the execution of the MG-RAST (and other) pipelines on a wide array of computational platforms. We have already provided cloud (aka Amazon EC2) machine images to a number of groups interested in providing their own computational resources for their data while analyzing the data with MG-RAST.

– More security

MG-RAST now supports encryption for your passwords. We note that the MG-RAST system does not offer secure connections (due to the increased hardware cost that would create).

************************************************
Analysis Pipeline slowdown
************************************************

You may have noticed that jobs submitted since June are taking longer to be processed. There are many contributing reasons — multiple hardware problems, move to the new Magellan cluster configuration, and preparing for the move to the SHOCK/AWE pipeline. In addition, there were some kinks in the new pipeline which are being worked out. The net result has been a large backlog in the compute queue and an increase in analysis time. Most of these issues have been resolved and we are working on resolving the last technical hurdles as quickly as we can.

************************************************
Metadata issue
************************************************

MG-RAST uses a controlled vocabulary for the metadata entries biome, feature, and material. This ontology, which was created and is controlled by EnvO (Environment Ontology), is available on the BioPortal site. Some terms from the latest EnvO version do not validate correctly on the MG-RAST website, resulting in an error message. We are revising our internal metadata validation process and until this is completed, you will need to select from compatible metadata terms from the list at:
http://api.metagenomics.anl.gov/1/metadata/cv?label=biome (or label=feature or label=material)
The list is returned as a JSON structure, use a viewer, e.g. FireFox + JSONView, to format it in a human-readable manner.

************************************************
Publishing MG-RAST IDs
************************************************

If you cite a MG-RAST ID in a publication you are responsible for making the data public on the MG-RAST site — all data in MG-RAST is private by default and has to be made public by explicit action of the owner. The declaration made during the submission process of the intention to make data public is only used to assign priority for the compute queue, your data will not be made public automatically based on the date entered.

The MG-RAST manual (ftp://ftp.metagenomics.anl.gov/data/manual/mg-rast-manual.pdf and linked from the front page) has more information in Section 4.11.

If you provide links to public datasets in a publication, use the linkin.cgi mechanism, not the URL displayed by the browser.
For example for the public dataset with MG-RAST ID 4440283.3 the linkin URL is:
http://metagenomics.anl.gov/linkin.cgi?metagenome=4440283.3
and for the public project with project ID 128 the linkin URL is:
http://metagenomics.anl.gov/linkin.cgi?project=128
These URLs provide a stable method to link to public datasets in MG-RAST.

************************************************
Citing MG-RAST
************************************************

If you use our service for analysis or to make your data public, please cite:
The Metagenomics RAST server — A public resource for the automatic phylogenetic and functional analysis of metagenomes
F. Meyer, D. Paarmann, M. D’Souza, R. Olson , E. M. Glass, M. Kubal, T. Paczian , A. Rodriguez, R. Stevens, A. Wilke, J. Wilkening, and R. A. Edwards
BMC Bioinformatics 2008, 9:386
http://www.biomedcentral.com/1471-2105/9/386

************************************************
UI Changes/Bug fixes
************************************************

FireFox v30 problems: upload broken, minor visual glitches
metadata file validation error for metadata files with integer metagenome names
collections removed from cacheing
fix missing list for jobs in progress
visual updates to job progress

Regards,
Folker Meyer
and the MG-RAST team

MG-RAST data migration, September 2, 2014

Tuesday, September 2nd, 2014

MG-RAST is moving from using a traditional file system to store data to an object management system SHOCK (https://github.com/MG-RAST/Shock). The data involved includes but is not limited to sequence files, intermediate analyses pipeline outputs and annotation products.

The change will take place in the backend and the website should not be impacted, all webpages and analyses should work normally. However, if you do notice a problem with the website please let us know as soon as possible with all relevant details.

This change will have an impact on the MG-RAST ftp site. While most public datasets will remain available in the short term, projects which have been made public recently will not be accessible at all. We are working on a permanent long-term solution to this problem and should have it in place shortly.

We thank you for your patience and understanding.

— the MG-RAST team