A dangerous $# in perl arrays - Not recommended for iteration

 The innocent looking $# operator that we often use for determining the maximum index of an array in for loop can be sometimes dangerous.

I spent a sizable amount of time wondering why my for loop is becoming an infinite loop without realizing that writing something like this actually changes the max index value or $# value of an array

Say you have 2 dimensional array @sorted and you want to print the component. The easiest way would be:

for(my $i=0; $i <= $#sorted; $i++){

                for(my $j=0; $j <= @{$sorted[$i]}; $j++){

                        print "$sorted[$i][$j]\t";

                        }

                        print "$#sorted\n";

        }

The output will be a neat:

3349    4097    gene-PR001_g8806                9 ----> The last column indicates the max index 

6662    6832    gene-PR001_g8807                9

11316   11696   gene-PR001_g8808                9

13158   13334   gene-PR001_g8809                9

18688   19095   gene-PR001_g8810                9

25175   25342   gene-PR001_g8811                9

26554   26883   gene-PR001_g8812                9

28100   29059   gene-PR001_g8813                9

29128   30235   gene-PR001_g8814                9

30266   30786   gene-PR001_g8815                9

Here one can notice the value of the last column is that of the max index of the array that remains unchanged and hence the lop terminates.

However, something as innocent as a c style this involving accessing the $i+1 element of the array actually changes  the array maximum index!! This came as a surprise to me where I hit the infinite loop leading to out of memory and locked file alert.

Check this out::

for(my $i=0; $i <= $#sorted; $i++){

                for(my $j=0; $j <= @{$sorted[$i]}; $j++){

                        print "$sorted[$i+1][$j]\t";---> Accessing the $i+1 value rather than $i value

                        }

                print "$#sorted\n";

                }

Here all the hell breaks loose where you hit a infinite loop when you suspect the least. Therefore, it

will be prudent to first pass the value of $#sorted to a variable and loop over that value instead of looping directly over $#sorted.

3349    4097    gene-PR001_g8806                9 

6662    6832    gene-PR001_g8807                9

11316   11696   gene-PR001_g8808                9

13158   13334   gene-PR001_g8809                9

18688   19095   gene-PR001_g8810                9

25175   25342   gene-PR001_g8811                9

26554   26883   gene-PR001_g8812                9

28100   29059   gene-PR001_g8813                9

29128   30235   gene-PR001_g8814                9

30266   30786   gene-PR001_g8815                9

10

11

... --> Increases infinitely!

A potentially dangerous infinite loop where you are least suspicious!!!

A neat solution for this problem will be:

my $index = $#sorted;--> notice this statement

        for(my $i=0; $i <= $index; $i++){

                for(my $j=0; $j <= @{$sorted[$i]}; $j++){

                        print "$sorted[$i+1][$j]\t";

                        }

                print "$#sorted\n";

                }

6662    6832    gene-PR001_g8807                9

11316   11696   gene-PR001_g8808                9

13158   13334   gene-PR001_g8809                9

18688   19095   gene-PR001_g8810                9

25175   25342   gene-PR001_g8811                9

26554   26883   gene-PR001_g8812                9

28100   29059   gene-PR001_g8813                9

29128   30235   gene-PR001_g8814                9

30266   30786   gene-PR001_g8815                9

10 --> Notice this 10 below. 

This means that the value of index is on rise but the loop terminates nevertheless!! Is this a bug in perl??

Piping server for transferring data back and forth between any device

 Today I came across a system called as piping server. The beauty of this technology is you can transfer any file between the devices using simple commands such as curl. If you are ubuntu or any other linux user, the files that you want to transfer from machine A to machine B involves the following simple commands.

Suppose in 'A' you have a file called as mandel1.jpg and you need to transfer that to 'B' then simply go to the terminal in A and type:

$ curl -T mandel1.jpg  https://ppng.io/mandel1

[The following will prompt in your terminal @ A]

[INFO] Waiting for 1 receiver(s)...

Then go to terminal 'B' and type:

sutripa@amrit:~$ curl https://ppng.io/mandel1 > mandel1.jpg

You will get the following prompt @B 

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100   137  100   137    0     0      2      0  0:01:08  0:01:06  0:00:02    33

Then finally do an ls at B

you will see the file you have transferred e.g; mandel1.jpg.

Transferring directories between devices is also pretty easy using this server.

At the sending server just do a 

$tar zfcp - ./QC | curl -T - https://ppng.io/such

or if you want to compress using zip do the following:

$zip -q -r - ./QC  | curl -T - https://ppng.io/such

At the receiving server do a 

sutripa@amrit:~$ curl  https://ppng.io/such > QC

Then you will see

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  855k    0  855k    0     0  60467      0 --:--:--  0:00:14 --:--:--  216k

Directory transferred.

For more information check this link out:

https://ostechnix.com/transfer-files-between-any-devices-using-piping-server/ 

two-speed genome analysis using R and perl

I have discussed about my speed genome analysis on my previous blog, now am writing the steps how to do that

1. calculate the intergenic distance of ur organism "x" from gtf file using the perl script , I have used the augustus predicted gtf file , the file needs to be modified according to the perl script so that it looks for the particular feature and pattern such as it looks for gene/exon/mRNA feature

sample of the augustus gtf file
 (make sure the last column is mentioned in the gene row)

scaffold_1904   AUGUSTUS        gene    1       775     0.09    +       .       transcript_id "g18705"; gene_id "temp";
scaffold_1      AUGUSTUS        transcript      1       4141    0.05    +       .       g1.t1
scaffold_1      AUGUSTUS        intron  1       180     0.55    +       .       transcript_id "g1.t1"; gene_id "g1";
scaffold_1      AUGUSTUS        intron  2022    3511    0.21    +       .       transcript_id "g1.t1"; gene_id "g1";
scaffold_1      AUGUSTUS        CDS     181     2021    0.09    +       2       transcript_id "g1.t1"; gene_id "g1";
scaffold_1      AUGUSTUS        CDS     3512    4141    0.49    +       0       transcript_id "g1.t1"; gene_id "g1";
scaffold_1      AUGUSTUS        stop_codon      4139    4141    .       +       0       transcript_id "g1.t1"; gene_id "g1";

use the perl script Calculate_FIR_length.pl using the gtf file to calculate the intergenic distance between the features, already complimentbed is there but that is not the one which we need, that is for difference purpose when I posted in a github forum to author I came to know the difference, this is the link https://github.com/Adamtaranto/density-Mapr/issues/1#issuecomment-291475238 

the intergenic distance between the gene file looks like this

"geneid","strand","fiveprime","threeprime"

"g10","+",5020,927

"g84","+",3316,8625

"g42","+",1558,1773

"g156","+",4460,13837

"g93","-",2035,553

"g30","+",117,361

"g106","+",1656,874

"g39","+",1380,720

"g1","+",NA,1222

"g70","+",2614,419

2. make a gtf file of Avh's with the location and calculate the intergenic distance for Avh's , intergenic 5 prime and 3 prime distance needs to be calculated the intergenic distance of effector file should look like this

"geneid","strand","fiveprime","threeprime"

"Avh92","-",4946,80942

"Avh48","+",137224,80942

"Avh102","+",38474,24067

"Avh127","-",137224,6955

"Avh304","-",7882,12043

"Avh313","+",23825,26166

"Avh91","-",24826,4946

"Avh303","+",16698,12043

"Avh34","-",41377,26166

"Avh61","+",4031,5317

"Avh311","-",14467,1021

"Avh310","+",4389,1021

"Avh93","+",41377,38474

3.  then just Run the R script which is pasted below just by changing the names of the file, if the points or plots are going out the cut off please change the bin size and num of bins before its creating the heatmap don't change after the heatmap is made

whole_intergene=read.csv(file="intergene_whole.csv",sep=",")

NumBins=50

if ((max(whole_intergene$fiveprime, na.rm=TRUE)>max(whole_intergene$threeprime, na.rm=TRUE)) == TRUE) { whole_intergene2Bin=whole_intergene$fiveprime} else { whole_intergene2Bin=whole_intergene$threeprime}

whole_intergene2Bin=whole_intergene2Bin[which(whole_intergene2Bin!=0)]

whole_intergene2Bin=na.omit(whole_intergene2Bin)

BinSteps=round(length(whole_intergene2Bin)/ (NumBins-20) , digits=10)

whole_intergene2BinOrd=sort(whole_intergene2Bin)

#### The [2*BinSteps] has been changed to [1*Binsteps it was producing an error after googling the error has been fixed]

TempBinLimits=whole_intergene2BinOrd[seq(whole_intergene2BinOrd[2*BinSteps],length(whole_intergene2BinOrd),BinSteps)]

TempBinLimits[length(TempBinLimits)+1]=max(whole_intergene2Bin, na.rm=TRUE)

x<-seq(length(TempBinLimits))

fit<-nls(log(TempBinLimits) ~ a*x + b, start= c(a=0, b=0),algorithm='port',weights=((x-1.0* NumBins)^2))

pred=predict(fit, x)

BinLimits=c(1, round(exp(pred),0), max(whole_intergene2Bin))

xbin=cut(whole_intergene$fiveprime, breaks=c(BinLimits))

ybin=cut(whole_intergene$threeprime, breaks=c(BinLimits))

whole_intergene=cbind(whole_intergene, xbin, ybin, genevalue=rep(1, length (whole_intergene$fiveprime)))

GenValMatrix<-with(whole_intergene, tapply (genevalue, list(xbin, ybin), sum))

x<-1:ncol(GenValMatrix)

y<-1:nrow(GenValMatrix)

zlim = range(as.numeric (unlist(GenValMatrix)) , finite=TRUE)

mypalette<-colorRampPalette(c( "white","darkblue", "forestgreen", "goldenrod1","orangered", "red3", "darkred"), space="rgb")

mycol=mypalette(2*max(GenValMatrix, na.rm=TRUE))

mylabels<-paste(BinLimits[1:length(BinLimits)-1], BinLimits[2:length(BinLimits)], sep="- ", collapse=NULL)

filled.contour(x, y, z=GenValMatrix,plot.title = title(main ="Phytophthora ramorum Pr102 genome",xlab = "five prime intergenic regions", ylab= "three prime intergenic regions", cex.main=0.8, cex.lab=0.8),key.title = title(main ="Number ofgenes", cex.main=0.5,line=1),col=mycol,levels = pretty(zlim, 1*max(GenValMatrix,na.rm=TRUE)),plot.axes={axis(1,at=x, labels=mylabels, las=2,cex.axis=0.5);axis(2,at=y, labels=mylabels,cex.axis=0.5)})

#wget http://wiki.cbr.washington.edu/qerm/sites/qerm/images/1/16/Filled.contour3.R

source('Filled.contour3.R')

library(png)

library(gridExtra)

library(ggplot2)

image_name<-paste(as.character(format(Sys.time(),"%Y%m%d%H%M%S")), "_graph", sep="")

png(filename = paste(image_name, ".png", sep=""))

par(mar=c(0,0,0,0))

filled.contour(x, y, z=GenValMatrix,col=mycol,levels = pretty(zlim, 2*max(GenValMatrix,na.rm=TRUE)),frame.plot = FALSE,axes = FALSE)

dev.off()

img <- readPNG(paste(image_name, ".png", sep=""))

library(gridExtra)

library(grid)

library(ggplot2)

library(lattice)

g <- rasterGrob(img, interpolate=TRUE)

rxlrData=as.data.frame(read.csv('rxrl_whole_intergenic.csv',header=TRUE))

ggplot(data=rxlrData,aes(x=rxlrData$fiveprime,y=rxlrData$threeprime,geom="blank"))+annotation_custom(g,xmin=-Inf,xmax=Inf,ymin=-Inf,ymax=Inf)+coord_fixed(ratio=1)+geom_point(shape=21,fill="red",colour="black",size=2,alpha=0.7,na.rm=FALSE)+scale_y_log10(breaks=BinLimits[2:length(BinLimits)],limits=c(BinLimits[2],BinLimits[NumBins +1]))+scale_x_log10(breaks= BinLimits[2:length(BinLimits)],limits= c(BinLimits[2] ,BinLimits[NumBins +1]))+theme(axis.text.y=element_text(size = 10,vjust=0.5))+theme(axis.text.x=element_text(size=10,vjust=0.5,angle=90))+theme(axis.title.x = element_text(face="bold",size=12))+xlab("five prime intergenic region")+theme(axis.title.y = element_text(face="bold",size=12)) + ylab("threeprime intergenic region")

Below I have put the screen shot for the Phytophthora sojae genome from a sample dataset

 minor edits can be done to make it good !!

This code and concept has been acquired from http://biorxiv.org/content/early/2015/07/01/021774 

analysing the alleles from haplotypes on pacbio data

I had been working in pacbio data and when am trying to identify the alleles from haplotypes from diploid assembly, in the very early step itself i got many errors, because i had been following the illumina dataset method like for pacbio data, but the developed tools behaves strange with the data and I got stuck for 3-4 days i googled the maximum and tried various approaches, Finally i posted in the forums and interacted with the GATK developers, they suggested me a simple solution for solving my errors, so those who are working in long reads and want to identify the haplotypes here is my commandline and verified one
[ Any aligners can be used even BLASR initially i was thinking there was a problem with my aligner, but really not] and no need to mark duplicates in case of long reads only for illumina reads its been recommended by the developer, i had reached till the step of HaplotypeCaller so far no error its running smooth, If i change commands or face any problems, will be updated, once the output is ready maybe i can paste some of my output

bwa index 2017_V6_Pr102_assembly.fa
bwa mem -x pacbio 2017_V6_Pr102_assembly.fa /data/results1/STLab/Takao_data/Raw_data/ND886/all_ND886.fastq > aln.sam
samtools view -b -S aln.sam -o aln.bam
samtools sort aln.bam > aln_sorted.bam
samtools index aln_sorted.bam
samtools mpileup -uf 2017_V2_ND886_assembly.fa aln_sorted.bam | /share/apps/bcftools-1.2/bcftools call  -cv - > out.vcf [use bcftools1.2 otherwise its not producing the genotype information]

Use any of your favourite haplotype phaser (whatshap/ hapcut) along with the above produced bam and vcf file

Now u get the phased  alleles from haplotypes u can compare them and these can be used to downstream analysis

Fancy genomics “Iam taking you all to the world of two-Speed genomes concept"

My Phd problem includes the various approaches for solving genome assembly problems. When I was working on oomycetes project, I was attracted by the effector proteins, Evolution, pathogenicity, synteny, transposon, Repeat regions, suddenly the fancy thing which came in the mind after reading an interesting paper from biorxiv that is verticullum genome, a group from Netherlands have sequenced and studied the 2-speed genome concepts among the strains. http://genome.cshlp.org/content/early/2016/07/12/gr.204974.116.full.pdf+html I was impressed by the work, then I showed the work to my PI even she was impressed by the speed genomes. I work in a collaborative program where exactly my collaborator also was fascinated by the  speed genome work.

Let me explain what is 2 speed genomes?

It was already known that fungi and the plant pathogen genomes comprises of Effector proteins. Which plays an important role in causing pathogenicity to the host. These Effector genes are not randomly distributed across the genomes, tend to be associated with the compartments enriched with repeat sequences and transposons. This led to the ‘two-speed genome’ model in which filamentous pathogen genomes have a bipartite architecture with gene sparse, repeat rich compartments for adaptive evolution.  The unusual genome architecture and occurrence of effector genes in specific genome compartments is a feature that has evolved repeatedly in independent phylogenetic lineages of filamentous pathogens. Genome analyses of P. infestans and three of its sister species revealed uneven evolutionary rates across genomes with genes in repeat-rich regions showing higher rates of structural polymorphisms and positive selection.  Two-speed genome architecture with the effector genes populating the more rapidly evolving sections of the genomes.  Lineages that acquired two-speed genomes have increased survivability — they are less probabe to go extinct compared to lineages with less adaptable genomes, which are more probabe to be purged out of the biota as their hosts develop full resistance or become extinct. In this ‘jump or die’ model, pathogen lineages that have an increased likelihood to produce virulent genotypes on resistant hosts and non-hosts benefit from a macroevolutionary advantage and end up dominating the biota. Several filamentous plant pathogens have evolved by shifting or jumping from one host plant to another.

The information has been shared from this paper a great detailed review by Sophien and Raffaele et al its available here http://www.sciencedirect.com/science/article/pii/S0959437X15000945 .

For who don’t have access to science direct the same paper is available at biorxiv repository please find the link http://biorxiv.org/content/early/2015/07/01/021774

Structural variation in the genomes

Structural variation:Structural variation is a change or variation which leads to change in the structure of organisms's chromosome. structural variants can be of Insertions, duplication, Inversion and translocation. According to the human genome or people work in genome say that if there is a variant more than of 50 base pairs changes in the human genome of 1%. Its believed that some of the genetic diseases are caused due to the structural variations.whats the difference between the SNP's and structural variation?SNP's are single nucleotide base mutations  which have been validated to be present in more than 1% of the population when a single base differes between the 2 genomes.  These are any mutations which cause a change in the organism's chromosome structure, such as Insertions, deletions, copy number variations, duplications, inversions and translocation.  SNPs and INDELs are about low-level genomic variation. The structural variants which affect the genome at larger scales. Events like gene duplications, tandem repeats, transposon insertions, inversions, and other chromosomal rearrangements. The long read sequencing technology paves the way to understand the structural variants using the split read alignment.[Information from literature Structural variation in two human genomes mapped at single-nucleotide resolution by whole genome de novo assembly  Yingrui Li, et al]  structural variations from short sequencing reads are hampered by one or more of the following limitations: (i) the methods may favor a particular length range of structural variations; (ii) they may favor discovery of particular types of structural variations; (iii) they may be unable to resolve the exact structural variation genotypes and/or breakpoints at single nucleotide resolution; and (iv) because of difficulties mapping reads to the genome, they may not be able to accurately identify complex rearrangements. Paired-end mapping, for example, can only predict insertion breakpoints within a few base pairs of the exact breakpoint position, and it can only detect insertions when the entire sequence is contained within the DNA fragment whose ends are being sequenced; thus, the maximum size of an insertion that can be detected by paired-end mapping is limited by the largest insert size present in a library. Split-read methods, on the other hand, can precisely define a breakpoint and genotype of an insertion, but only when it is shorter than the read length. Thus, studies carried out so far have been of limited completeness, accuracy and/or resolution.

BWA-MEM or BLASR 

http://lh3.github.io/2014/12/10/bwa-mem-for-long-error-prone-reads/ this is a very nice blog discusses about the alignment methods useful of the pacbio long reads. 

https://www.biostars.org/p/63306/  forum discusses about the split read alignments.

Tips for structural variant analysis:

1. The maximum number of Reads should be mapped in the breakpoints of the chromosome and the coverage should be high.

2. How many Individual reads are supporting the translocation versus supporting assembly for identifying the translocations.

[ I spoke with some of the developers asking about the structural variants of draft pacbio assembly plant pathogen human said completely I can use the tools for predicting , am trying to do for one of the plant pathogen genome]

one of the paper in 2014 talks about all approaches

https://bib.oxfordjournals.org/content/early/2014/12/12/bib.bbu047.full#sec-9

Posters from ECCB2016

I found some interesting poster and thought it will help my friends who are working on the same area , and same type of work going in my lab those are here

ECCB 2016 Den Hague, Netherlands computational Biologist and Bioinformaticians gatherings at a sweet Dutch country !

I have been to several conferences within India, while ECCB 2016 which happened in Den Hague, from September 3 2016 - September 7 2016. It was the first time for me travel outside India, had butterflies on my stomach the day before I travel.The trip really went well. It was a gathering of computational Biologist and Bioinformaticians over the world. Well I should thank Department of Science and Technology, Government of India for providing me the travel award. The Meeting started with the workshop on discussing Pacbio and Nanopore data. Expertise from the field of nanopore and Pacbio were discussing the problems with the long reads. People were complaining about the "error rates" of these reads, and difficulties in genome assembly of these reads. Had a great opportunity to discuss with the experts. The Nanopore experts were suggesting that Canu assembler can do better when handling the problematic regions in the genome. The Miniasm and Racon assembler also be tried . There were sessions about Irys to create a genome map and align the created  map back to the genome assembly to get a better genome assembly. The structural variants and the comparative genomics are also studied from the graph. Next topic was using Isoseq from pacific Bio-systems to produce a full length transcripts without assembly, followed by promethion and squiggle sequencing system from nanopore technology"Read Until " approach it enables selection of individual DNA molecules for sequencing from a pool of DNA molecules. Then there was a session of Minotour where the base calling of the nanopore reads where done without performing the cloud base calling since there is a dependency of high speed internet. The developers of the tools and technologies were very friendly and gave suggestions on working with the long reads. after the workshop the conference scientific sessions started and many interesting talks where there, I was more interested towards the error correction algorithm development, genome assembly tools, new ortholog prediction tools. Most of the sessions and posters where about the cancer ( a devil), and ENCODE. I can say that 60% of people presented towards cancer transcriptomics and genomics, and 30 % of work in ENCODE, rest where like plant, bacteria, database development, Docking and simulations. The talks and discussions can be retrieved from twitter via #ECCB2016.  I liked the theme of the conference here not only PhD students and Scientists were presenting the work, even people working in companies were also showcased the ongoing work.Some people were very happy and showed interest towards the  poster of my PhD work, since its a plant pathogen. I am more interested towards studying the environmental organisms, pathogens of human, discovering various new species from the environment. About the food it was good, had a varieties of cheese. I had time to visit Amsterdam its a very nice place with a polite people. Visited churches, Museum, had a good canal Boat riding.I had few friends from conference and joined with them and rented a boat and rode over the Amsterdam city. The future ECCB2017 conference will be held in Prague.

Analyzing Differential expression analysis data using the tuxedo suite (cummeRbund)

Tuxedo suite comprises of bowtie, tophat, cufflink, cummeRBund and many more accessory tools.

First get your genome fasta file (final genome assembly file).
1. Map your RNAseq fastq files using tophat (if all is well your run will be seamless)
2. Run cufflink over your tophat output file (cufflinks accepted_hits.bam). This run will take a while since cufflink will actually merge the reads into transcripts, isoforms, genes and so on. If your files are large then in a good enough server expect it to run for 8-12 hours.
3. Run cuffmerge: cuffmerge list.txt -> where list.txt carries the names of the files of *_transcripts.gtf files. This will run very fast and will merge all the gene_ids that will be same across all your samples. The output of this file is a merged.gtf file.
4. For running differential expression analysis run the following:
/cuffdiff merged.gtf tophat_HTI1-vs-HTI4/accepted_hits.bam tophat_HTI2-vs-HTI4/accepted_hits.bam tophat_HTI3-vs-HTI4/accepted_hits.bam

This will create a plethora of files, but the following files are the ones you will be proceeding with for cummeRbund for result visualization and generating publication quality images.

For running cummeRbund, get all these files to your working directory

isoforms.fpkm_tracking

isoform_exp.diff

genes.fpkm_tracking

gene_exp.diff

tss_groups.fpkm_tracking

tss_group_exp.diff

cds.fpkm_tracking

cds_exp.diff

cds.diff

promoters.diff

splicing.diff

The best option will be to put all of these 11 files into a separate directory inside your working directory: say 'diff_exp'
You can run Rstudio if you like in your windows machine or run R in your server. For running CummeRbund you will need the following packages that you can go ahead and download upfront:

• RSQLite
• ggplot2 v0.9.2
• reshape2
• plyr
• fastcluster
• rtracklayer
• Gviz
• BiocGenerics (>=0.3.2)
• Hmisc

In case you have forgotten how to install R packages go this way: source('http://www.bioconductor.org/biocLite.R') biocLite('cummeRbund') And follow this same protocol for installing other R packages. Once done you can start with setting your working directory using setwd() command.
 For example: setwd("C:/Users/Sucheta/Documents/MyLabIICB/AllCollaborations/NahidAliCollaboration/companion")
Then load the library:

library(cummeRbund)

Now read your 11 files using this command
data <-readCufflinks("diff_exp")

This will take a while to read but will create a db file in your source directory. This is your database file.

Now you can plot gene density using the following command:

csDensity(genes(data))

Or can do a volcano plot of differentially expressed genes using:

v<-csVolcanoMatrix(genes(data))
v

As you can see from this file, the different conditions have least difference among themselves.

This will continue in next blog...

#OMGN2016 Malmo, Sweden - Between Then and Now...

Many things have changed in the years in front of me since the day I started attending OMGN meetings. My first meeting was in year 2005 and then the first Oomycetes genomes were getting sequenced and getting analyzed - at its own pace (read very slow pace). We used to get excited even when we got SSRs or repeats predicted. I distinctly remember the 2004 Joint Genome Institute sequence jamboree when in the evenings we used to gather to discuss what was done during the day. On second or third day of Jamboree, Brett came up with this multiple sequence alignment that presumably indicated that there was an RXLR motif in the effector proteins. It was a huge deal then. Subsequently in all the meetings everybody started discussing on these proteins. Initially it appeared too good to be true with this small 4 letter motif, but a lot of work was done especially in Brett's lab to prove that it indeed was a significant motif. The prediction algorithms of effectors got published in high flying journals, everybody was excited. Slowly many more papers came out on RXLRs, their prediction methods, characterizations till 2010. Now in 2016, I see the level of science has gone up way higher. Genome sequencing using PacBio or Illumina is no deal, neither is analyzing them. Effector prediction has become just a days job (Thanks to all the hard work done by the pioneers). Genome analysis are now carried out by single individuals in few months time. This meeting was a skew towards miRNAs, CRISPER technology for genome editing, pacbio sequencing, RNA silencing. The effector biology has moved many many steps up now. Many more things are now known. Many more proteins have been characterized. It is exciting time in the history of oomycetes biology where many things are happening right in front of our eyes. For those who could not attend please check #OMGN16 for more details. For me now bye bye lovely Malmo!!

Variant Calling - The bowtie - picard - samtools - gatk pipeline....

Nextgen sequencing has caused a sudden surge in data deluge, but the informatics pipelines and algorithms are unable to keep up with the pace. While most of the exome sequencing data finally focuses on SNP calling and there are various ways of doing this, I decided to discuss one pipeline that has been accepted all over as one of the most sophisticated methods. It is the bowtie - picard - gatk pipeline.

When you are dealing with colorspace data the choice of mappers get limited. Howver, my favorite mapper is still bowtie for several reasons. Lifescope has its own inhouse mapper; which claims to have a all round better approach in mapping colorspace data, but the lack of transparency on what happens within puts me off using this tool. Once bowtie maps the reads by default parameter, the next thing to do is to convert the sam file into bam file, sort it and index it. All these can be done using samtools. However, if the file size is large, you could do the sorting job using sort operations from unix commandline.

1. sort sam file
export TMPDIR=DIR_WITH_LOTS_OF_SPACE
LC_ALL="C" sort -k 3,3 -k 4,4n input_sam > output_sam # This step will take a long time

sort options for samtools works but only on bam files and on many instances downstream analysis softwares complain about co-ordinates not being sorted...

or Use picard:

java -jar /share/apps/picard-tools-1.56/SortSam.jar I=bowtie.sam O=bowtie.bam SO=coordinate # This took one hour in a HPC with 48 GB RAM on each node for a file size of 30 GB

2. Make an index file of bam file
samtools index bowtie.bam bowtie.bai

3. MarkDuplicates using picard
java -jar /share/apps/picard-tools-1.56/MarkDuplicates.jar I=bowtie.bam M=metrics.bam O=duplicateMarked.bam

4. sort this bam file and make index using samtools
samtools sort duplicateMarked.bam duplicateMarked.sorted
samtools index duplicateMarked.sorted.bam duplicateMarked.sorted.bai

5. Then run IndelRealignerTargetCreator using GATK
java -jar /share/apps/GenomeAnalysisTK-2.4-9-g532efad/resources/GenomeAnalysisTK.jar -T RealignerTargetCreator -I duplicateMarked.sorted.bam  -R /share/reference/human/samtools/hg19.fa -o dM.bam.list 
# The output file dM.bam.list returns 0 output. Check it later.

6. Now run this picard tool to get RG updated since indelRealigner complains.

java -jar /share/apps/picard-tools-1.56/AddOrReplaceReadGroups.jar I=duplicateMarked.bam O=readGroupReplaced.bam RGLB="LINK_TO_FASTA" RGPL=SOLID RGPU=run barcode RGSM=9111 SORT_ORDER=coordinate CREATE_INDEX=TRUE VALIDATION_STRINGENCY=LENIENT'

Posters I could take pictures of Beyond Genome 2014

Here are few of the posters in Beyond Genome meeting that I could take pictures of. There were many more, but access to take their pictures was less...

Our Poster

  

SNP calling using GATK for de novo genome

I have a got a chance to work in leishmania genome, where i have a genome assembly and i dont have any deposited dbSNP or any other reference file to do variant calling, i have been working and stuck in many steps and posted in GATK forums they replied to some of my queries at one point stopped to reply since People were having a fat and busy holiday on thanks giving , and figured out how to do the variant calling, i think this blog will be much useful for the naive person like me, lets see the workflow and please refer GATK documentation for the explanation.
#first build the index for the reference genome
/share/apps/bowtie2-2.1.0/bowtie2-build after_removing_2k.fasta  leishmania.index.bt2
#after index map the reference to the reads
/share/apps/bowtie2-2.1.0/bowtie2 -x leishmania.index.bt2 -1 /data/results/STLab/NahidAli/141218_SND393_A_L005_HTI-5_trim_R1_filtered.fastq -2 /data/results/STLab/NahidAli/141218_SND393_A_L005_HTI-5_trim_R2_filtered.fastq -S bowtie_aligned.sam
#convert the sam file to bam
samtools view -S bowtie_aligned.sam -b -o bowtie_aligned.bam
#sort the bam file
samtools sort bowtie_aligned.bam bowtie_aligned_sorted
#create a pileup file
samtools mpileup -uf after_removing_2k.fasta bowtie_aligned_sorted.bam|/share/apps/samtools-0.1.18/bcftools/bcftools view -bvcg - > leishmania.raw.bcf
#convert bcf to vcf
/share/apps/samtools-0.1.18/bcftools/bcftools view leishmania.raw.bcf > leishmania.raw.vcf
*********************************************************************************

 The bove commands are just initial way of mapping the reads to the reference and the real GATK pipeline starts below since i don't have the any known sites  i have done without base cailbration 

#create the dictionary
#java -jar /share/apps/picard-tools-1.56/CreateSequenceDictionary.jar R=after_removing_2k.fasta O=after_removing_2k.dict
#add or mark group ids
#java -jar /share/apps/picard-tools-1.56/AddOrReplaceReadGroups.jar I=bowtie_aligned.sam O=group_added_read.bam SO=coordinate RGID=1 RGLB=library1 RGPL=illumina RGPU=1 RGSM=leishmania VALIDATION_STRINGENCY=LENIENT CREATE_INDEX=TRUE
#mark duplicates
#java -jar /share/apps/picard-tools-1.56/MarkDuplicates.jar I=group_added_read.bam O=mapped_reads_dup.bam METRICS_FILE=metricsFile CREATE_INDEX=true
#sort bam file
#java -jar /share/apps/picard-tools-1.56/BuildBamIndex.jar INPUT=mapped_reads_dup.bam
#create realign target creator
#share/apps/GenomeAnalysisTK.jar -T RealignerTargetCreator -R after_removing_2k.fasta -o target_interval.intervals -I mapped_reads_dup.bam
#indel realigner
#/share/apps/GenomeAnalysisTK.jar -T IndelRealigner -R after_removing_2k.fasta -I mapped_reads_dup.bam -targetIntervals target_interval.intervals -o Indel_realigned.bam
#haplotype caller
#java -jar /share/apps/GenomeAnalysisTK.jar -T HaplotypeCaller -R after_removing_2k.fasta -I Indel_realigned.bam -stand_call_conf 30 -stand_emit_conf 10 -o raw_variants.vcf
#choose the variants from the raw vcf file
#java -jar /share/apps/GenomeAnalysisTK.jar -T SelectVariants -R after_removing_2k.fasta -V raw_variants.vcf -selectType SNP -o raw_snps.vcf
#do the filtration
#java -jar /share/apps/GenomeAnalysisTK.jar -T VariantFiltration -R after_removing_2k.fasta -V raw_variants.vcf --filterExpression "QD < 2.0 || FS > 60.0 || MQ < 40.0 || MQRankSum < -12.5 || ReadPosRankSum < -8.0" --filterName "my_snp_filter" -o filtered_snps.vcf
#Extract the indels
#java -jar /share/apps/GenomeAnalysisTK.jar -T SelectVariants -R after_removing_2k.fasta -V raw_variants.vcf -selectType INDEL -o raw_indels.vcf
#do the filteration
#java -jar /share/apps/GenomeAnalysisTK.jar -T VariantFiltration -R after_removing_2k.fasta -V raw_variants.vcf --filterExpression "QD < 2.0 || FS > 200.0 || ReadPosRankSum < -20.0" --filterName "my_indel_filter" -o filtered_indels.vcf

from the above predicted snps and indels extract the regions and further annotate and work on it happy variant calling !!!!!!!!!!

5C bed file data format

5C and 3C are the newer technologies in sequencing where the chromatin inetraction data can be obtained. If you looking for such data and happen to download from UCSC genome browser, it may be hard to look around for format describing the fields. We asked the authors and here is the explanation:

The site from which you may download data may be this: https://www.encodeproject.org/experiments/ENCSR000CYD/

BED  file format descrition can be found from : https://genome.ucsc.edu/FAQ/FAQformat.html#format1 

Here is a sample data for GM12878 cell line:

chr22   31998728        33247041        5C_301_ENm004_FOR_292.5C_301_ENm004_REV_

32      1000    .       31998728        33247041        0       2       12744,40

98,     0,1244215,

chr5    131346229       132145236       5C_299_ENm002_FOR_241.5C_299_ENm002_REV_

33      1000    .       131346229       132145236       0       2       2609,210

5,      0,796902,

col1: Chromosome name

col2: Chromosome start

col3: chromosome end

col4: Name of the interacting sites (primer names)

col5:

col7: chromosome start

col8: chromosome end

col11: block sizes in comma separated list

col12: block offset in comma separated list

Now I will explain what col11 and col12 means...

the beginning of interacting site is the cromosome start and the beginning of offset is 0.

So, the interacting site begins at 31998728 + 0 and the interacting block length is 12744.

The beginning position of interacting site 2 is: 31998728 + 1244215 = 33242943

 The size of interacting block 2 is 4098. so, end of interacting site is 33242943 + 4098 = 33247041.

Here is a diagrammatic representation:

Algal Biotechnology Workshop at IIT Mumbai on 21st Nov 2015

It was an insightful workshop on algal biotechnology at VMCC hall, IIT Mumbai during 21st November 2015. The organizers managed to have the world leaders as speakers in this area. The workshop started with handing over the materials to the participants followed by welcome address by Dr. Wangikar from IIT Mumbai followed by an insightful talk by Dr. Santanu Dasgupta from Reliance Industries.

Summaries of some of the interesting talks are discussed here:

De. Duu-long Jee from Department of Chemical Engineering, National Taiwan University:
Lutein, one of the 600 naturally occurring carotenoids is abundantly found in marigold flowers as well as in Micro-algae. Dr. jee presented an overview of cost-effectiveness of Lutein production with microalgae vs marigold flowers. Although microalgae produces about 3-4 times more lutein compared to Marigold flowers, but the energy required to extract those from micro-alga makes it an expansive option. Marigold on the other hand needs less nutrient, less power but more space... So, there is a need for engineering micro-algae that can have enhanced Lutein production with lesser energy dependence for extraction.

John Beardall, Monash University:

Extremophiles will play a major role in algal biotechnology, since they have altered metabolism. It is a well known fact that CO2 is sequestered in algae to enhance growth. But growth and lipid accumulation are oxymoron. Don't happen at the same time. They have explored media as a way for determining what favors the optimized fatty acid production. Their observation indicates that some micro-algae grow really well in media with altered  source of carbon (glycerol and xylose) and also produce optimal fatty acid. Myxotrophic growth is favored for higher fatty acid production.

Jo-Shu Chang, Department of chemical engineering, National Cheng Kung University, Tanian, Taiwan:

Talked about CO2 sequestration by micro-algae and production of economically important compounds. He discussed about major energy components from micro-algae as Butanol, ethanol, H2, Diols, lactic acids and succinic acids. The effluent gas composed of 23.1% CO2, SOx 85 ppm, NOx 75 ppm and at temperature of 230C can be used for growing microalgae. Burkholderia (a proteo bacter) can be used for lipase production.

Min S. Park, Advanced Biomass R & D Center, BioEnergy Engineering and Research Laboratories, Dept. of chemical and Biomolecular Engineering, Daejeon, Republic of Korea:

Nanocloropsis is the choicest microalgae used for studying bioenergy production. These organisms have lipid droplets in their chloroplast. They have done series of signalling work involving Nanochloropsis and came to conclusion that JNK type of MAPK was highly activated under osmotic stress. NaCl induces osmotic stress -> acts upon MAPKK -> acts on MAPK -> represses Transcription factor -> inducing lipid production. They also observed that lipid production is inhibited by treatment of MEK specific inhibitor. The microbial culture community comprising the treatment plants mostly contained scenedesmus, Golenkinia, Microspora, Micractinium etc.

Jong Moon Park, Department of Chemical Engineering, School of Environmental Science and Engineering, Division of Advanced Nuclear Engineering, POSTECH, Republic of Korea:

He presented 2 different aspects of Bio-enegy production: 1. Enhanced fatty acid production from microalgae and ethanol production of Cyanobacteria.
In Cyanobacteria, they have used several approaches for enhancing ethanol production directly by manipulating few enzymes. One is glucose-6-phosphate 1-dehydrogenase, encoded by zwf and the other is Pdc.  His admission is that ethanol from these engineered bacteria is released out of the cell and hence is not dangerous for the organism itself.
His notable work is also on microalgae where they have used food waste water and municipal sludge as one of the combinations for optimal growth of microalgae. He has also suggested that the municipality wate or food waste water can be diluted 20 times for growing micro-algae in them.
Chlorella was used for bio-diesel production.
Article look up are: Dexter and Fu, 2009; Li, C. 2015 for ethanol from Cyanobacteria.

Apart from this there were many more interesting talks, that I am not delving upon here. So, in all, everyone is looking for a breakthrough in growing these organisms faster and producing fatty acids quickly....

Installing R packages that use shared library in Linux

Many R packages use scripts (or libraries) written in other languages like C, FORTRAN etc from shared libraries. Normally the main scripts (and their dependencies like header files(.h files)) are kept in the src directory inside the package. During installation of the package from the source file(.gz) using R CMD INSTALL somepackage.tar.gz, the scripts are compiled and generates some shared objects in the local directory which dynamically links to the shared library (to libsomename.so.some_number file) which is generally /usr/local/lib. This linking happens through some configuration file (/etc/ld.so.conf ) and some environment variables (e.g. LD_LIBRARY_PATH). Often the conf file and the environment variable does not contain the path of the shared library (mostly happens when users use their own shared library instead of the default) and thus during installation it shows the error:  "shared object not found.... no such file or directory".

one way to solve this problem is problem is to run the ldconfig (or /sbin/ldconfig) commands(preferably in verbose mode(-v) ).  This program creates the required links and cache to the most recent shared libraries.

example:

I faced similar type of error "shared object not found.... no such file or directory" during installation of the package fftwtools (R CMD INSTALL fftwtools.tar.gz). The steps I followed to fix the problem are:

1. error obsereved : can not open .../fftwtools/src/fftwtools.so: ... libfftw3.so.3...  no such file or directory.

2. located the file using:  locate libfftw3.so.3 (to be sure that the file exists)

output: 

/usr/local/lib/libfftw3.so.3

/usr/local/lib/libfftw3.so.3.3.2

3. run /sbin/ldconfig -v

output:

/sbin/ldconfig: Path `/lib64' given more than once

/sbin/ldconfig: Path `/usr/lib64' given more than once

........................

/opt/bio/EMBOSS/lib:

        libajax.so.6 -> libajax.so.6.0.3

        libnucleus.so.6 -> libnucleus.so.6.0.3

.................

/lib:

        libdevmapper-event.so.1.02 -> libdevmapper-event.so.1.02

        libiw.so.28 -> libiw.so.28

.......................

/lib64:

        libdevmapper-event.so.1.02 -> libdevmapper-event.so.1.02

        libiw.so.28 -> libiw.so.28

.........................

/usr/local/lib:

        libnucleus.so.6 -> libnucleus.so.6.0.5

        libeplplot.so.3 -> libeplplot.so.3.2.7

.........

        libfftw3.so.3 -> libfftw3.so.3.3.2

        libezlib.so.1 -> libezlib.so.1.1.0

.........................

4. Install the package:  R CMD INSTALL fftwtools.tar.gz

Hope the steps works for you. I will be happy to answer any queries regarding this issue. Thanks a lot for reading the post.