cclib-devel Mailing List for cclib

This and other features are needed for a web app I've started to write. I'm 
thinking more or less along the lines of what you (Noel) started in 2007 (see 
the attached emails)... an interface where anyone can point to a remote file, 
upload one, or copy-paste the log, and get a nice print out of the parsed 
data.

The basic infrastructure is working, with no whistles yet though. It just 
let's you specify a file and parses. Go ahead, try it out:
http://www.mmqc.org/cclib/

I'd like to go further after getting the parser interface running and 
published, and start up a database of log files and parsed data. I think the 
idea could be useful in many ways.

Cheers,
Karol

On Monday 20 July 2009 20:35:31 Noel O'Boyle wrote:
> Sounds interesting. I'm away at the moment, but will spend some time
> on cclib when I get back.
>
> - Noel
>
> 2009/7/18 Karol M. Langner <kar...@gm...>:
> > Hi,
> >
> >  Today in r855 I added a few lines to utils.py that retrieve a file via
> > HTTP and parse it. Here is how it looks like:
> > http://cclib.sourceforge.net/wiki/index.php/Using_cclib#Remote_files
> >
> >  Python is so fun!
> >
> > Cheers,
> > Karol

-- 
written by Karol Langner
Mon Jul 20 21:08:21 CEST 2009

Sounds interesting. I'm away at the moment, but will spend some time
on cclib when I get back.

- Noel

2009/7/18 Karol M. Langner <kar...@gm...>:
> Hi,
>
>  Today in r855 I added a few lines to utils.py that retrieve a file via HTTP
> and parse it. Here is how it looks like:
> http://cclib.sourceforge.net/wiki/index.php/Using_cclib#Remote_files
>
>  Python is so fun!
>
> Cheers,
> Karol
>
> --
> written by Karol Langner
> Sat Jul 18 13:28:21 CEST 2009
>
> ------------------------------------------------------------------------------
> Enter the BlackBerry Developer Challenge
> This is your chance to win up to $100,000 in prizes! For a limited time,
> vendors submitting new applications to BlackBerry App World(TM) will have
> the opportunity to enter the BlackBerry Developer Challenge. See full prize
> details at: http://p.sf.net/sfu/Challenge
> _______________________________________________
> cclib-devel mailing list
> ccl...@li...
> https://lists.sourceforge.net/lists/listinfo/cclib-devel
>

2009/7/20 Karol M. Langner <kar...@gm...>:
> On Friday 12 June 2009 08:36:43 Manuel Melle Franco wrote:
>> On Fri, Jun 12, 2009 at 12:09 AM, Karol Langner
> <kar...@gm...>wrote:
>> > Hi Manuel,
>> >
>> >  As far as I know, ONIOM is not implemented in an other quantum chem
>> > program
>> > that we support at the moment. That being said, parsing ONIOM-specific
>> > stuff
>> > from only Gaussian would not be very useful in general, as we aim for
>> > extracted the same data from many programs. Correct me if I'm wrong,
>> > Noel.
>> >
>> >  However, if ONIOM output is not parsed by cclib, then that is a bug and
>> > we should be able to fix it.
>>
>> Dear Karol and Noel,
>>
>> As far as I know only gaussian and nwchem have it.
>> I am quite new to it and currently doing the parsing myself (fgrep mostly)
>> but it works only in the cases I use,
>> and it is a bit of a pain in the ass to parse.
>>
>> I agree with Karol, I think it might not be easy to write a general parser
>> for oniom so that it would not be worth the effort to add it to cclib, at
>> list at this point, maybe when more program have it, if that ever happens.
>>
>> best regards
>>
>> Manuel
>
> Manuel,
>
>  I fixed the gaussian parser so that it does not fail now when parsing ONIOM
> calcs. Attached are two Gausian03 test files that I used for testing. Since
> it is not a new feature, I'm not adding them as tests or regressions.

Please do add the files - this is what the regression test suite is for. :-)

>  Please be aware that this does not mean that cclib now parses ONIOM output --
> it does not! It merely does not crash now. Perhaps some of the parsed data
> can be of use, anyway.


>  If you have output that still crashes the parser (using today's revision),
> please send it to us so we can make the parser more failure-proof.
>
> Cheers,
> Karol
>
> --
> written by Karol Langner
> Mon Jul 20 17:23:26 CEST 2009
>

On Friday 12 June 2009 08:36:43 Manuel Melle Franco wrote:
> On Fri, Jun 12, 2009 at 12:09 AM, Karol Langner 
<kar...@gm...>wrote:
> > Hi Manuel,
> >
> >  As far as I know, ONIOM is not implemented in an other quantum chem
> > program
> > that we support at the moment. That being said, parsing ONIOM-specific
> > stuff
> > from only Gaussian would not be very useful in general, as we aim for
> > extracted the same data from many programs. Correct me if I'm wrong,
> > Noel.
> >
> >  However, if ONIOM output is not parsed by cclib, then that is a bug and
> > we should be able to fix it.
>
> Dear Karol and Noel,
>
> As far as I know only gaussian and nwchem have it.
> I am quite new to it and currently doing the parsing myself (fgrep mostly)
> but it works only in the cases I use,
> and it is a bit of a pain in the ass to parse.
>
> I agree with Karol, I think it might not be easy to write a general parser
> for oniom so that it would not be worth the effort to add it to cclib, at
> list at this point, maybe when more program have it, if that ever happens.
>
> best regards
>
> Manuel

Manuel,

 I fixed the gaussian parser so that it does not fail now when parsing ONIOM 
calcs. Attached are two Gausian03 test files that I used for testing. Since 
it is not a new feature, I'm not adding them as tests or regressions.

 Please be aware that this does not mean that cclib now parses ONIOM output -- 
it does not! It merely does not crash now. Perhaps some of the parsed data 
can be of use, anyway.

 If you have output that still crashes the parser (using today's revision), 
please send it to us so we can make the parser more failure-proof.

Cheers,
Karol 

-- 
written by Karol Langner
Mon Jul 20 17:23:26 CEST 2009

Hi,

 Today in r855 I added a few lines to utils.py that retrieve a file via HTTP 
and parse it. Here is how it looks like:
http://cclib.sourceforge.net/wiki/index.php/Using_cclib#Remote_files

 Python is so fun!

Cheers,
Karol

-- 
written by Karol Langner
Sat Jul 18 13:28:21 CEST 2009

Hi

thanks for the changes to cclib!

> > In a related project I'm currently working on, I have already written
> > code to convert from forces in cartesians to forces in z-matrix, and it
> > numerically differentiates the function Z(C) that returns z-matrix
> > coordinates, Z, in terms of the cartesians, C, and implements the
> > function Z using the python interface to openbabel.
>
> That code would be a good starting point for these helper methods. Would
> you care to share it?

Sure, when it's in better shape I'll send it through.

I've also re-implemented the z-matrix => cartesian conversion (that is needed 
for the above) from OpenBabel in pure Python (Reasons: the openbabel version 
had bugs, didn't emit xyz with sufficient precision to do numerical 
differentiation, it required a lot of code to utilise it from within Python ). 
I can make both of these available if you want.

> > Well if I were to write some code to parse them, would you want to add it
> > to cclib?
>
> Perhaps. To start off with, we could create a branch of cclib that would
> implement just the gradients for VASP, and when we have enough attributes
> parsed compared to other programs, we could include it in the main
> development branch.

Ok. I will be doing some hacking over the coming months and as I have support 
implemented I'll send it through.

Cheers
Hugh

Oops... forgot to answer to the list.

----------  Forwarded Message  ----------

Subject: Re: [cclib-devel] Partial files
Date: Thursday 16 April 2009
From: Karol Langner <kar...@gm...>
To: "Noel O'Boyle" <bao...@gm...>

Hi,

 Sorry for answering with delay. I actually have given this some thought
before, and no, it's not necessary. You are correct to write that there are
too many possibilities to cover in the parsers. I will not go on adding
shredded output to the regression tests :)

 With that said, I still think that covering the most obvious/common cases
is worth it. The way I truncate output files sometimes (to save disk space)
is probably pretty typical... cut out the chunky sections such as
eigenvectors and populations. It's pretty important to me that these output
files can be parsed, even if they contain incomplete data.

 Also, the changes I made to the code that fix this are more general - try
clauses like the one you cited below. They only envelop a fragment of the
parser, so if something goes wrong there the error message is quite
specific.

Hope that's OK,
Karol

On Tue, Apr 7, 2009 at 10:48 AM, Noel O'Boyle <bao...@gm...> wrote:

> Karol,
>
> Is it really necessary to add warnings for partial output files? There are
> about 1000 points of failure here. What I do in GaussSum is a simple "catch
> all":
>
> try:
>    # parse with cclib
> except:
>    # message to user saying it couldn't be parsed, please contact support
> if necessary
>
> - Noel
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by:
> High Quality Requirements in a Collaborative Environment.
> Download a free trial of Rational Requirements Composer Now!
> http://p.sf.net/sfu/www-ibm-com
> _______________________________________________
> cclib-devel mailing list
> ccl...@li...
> https://lists.sourceforge.net/lists/listinfo/cclib-devel
>
>

-------------------------------------------------------

-- 
written by Karol Langner
Wed Jul 15 18:05:28 CEST 2009

On Tuesday 05 May 2009 12:50:33 Noel O'Boyle wrote:
> Apparently cclib is going into Fedora along with GaussSum and QMForge.
> While this is good news, it puts some pressure on us to keep the API
> stable.
>
> - Noel

Shouldn't we be getting into Debian by now, too? :)
I actually have cclib packaged into a deb already.

-- 
written by Karol Langner
Wed Jul 15 18:03:50 CEST 2009

Hi,

 I just "fixed" a couple of unit tests to use numpy.issubdtype instead of the 
generic isinstance of Python, but only when checking scalar types in numpy 
arrays. This is needed for recent versions of numpy, but I'm not sure when 
they added that method.

 I have numpy 1.2.1. Does anyone use an older version, where the tests don't 
run now? If so, we'll have to find a work-around for this.

- Karol

-- 
written by Karol Langner
Wed Jul 15 18:00:07 CEST 2009

To answer a few more questions...

> > Z-matrix parsing and/or interconversion with cartesians would be
> > useful... currently I do it on a regular basis with openbabel. If not
> > parsed, perhaps a helper method could build and return the z-matrix via
> > openbabel.
>
> That would surely be useful. Coul I quickly ask, do you use the C++ API
> directly when doing this or do you mean you convert between file formats
> with z-matrix/cartesian specifications?

No, I hack away in bash... passing around *.xyz files and text files with 
Z-matrices :) I also wrote a little function for converting between these 
things (in Python), but don't use it all that often it turns out.

> > For derived quantities like (second) derivatives, it will be harder to
> > maintain clarity and consistency between outputs from various programs if
> > we were to provide them potentially in both cartesian and internal
> > coordinates. I would again suggest a helper method in this case that
> > converts the standard cartesian form to internal coordinates, with a
> > Z-matrix or atom ordering given as input.
>
> Yep I suppose that would be good. When gradients are already available in
> terms of the z-matrix, I would have thought it incurs unnecessary
> complexity to invoke other methods to convert the gradients from cartesians
> to z-matrix. But in general, such infrastructure in the code would be
> useful.

Yes... in my opinion this should be done post-parsing, as a helper method. 
Cartesians and internals are fully interchangeable (besides the origin), so 
this really is not a problem, given the proper methods.

Note one more thing. Gaussian itself prints the internal gradient only when 
explicitly given Z-matrix input. It always print cartesian gradients, 
however, so just parsing that makes everything a lot simpler.

> In a related project I'm currently working on, I have already written code
> to convert from forces in cartesians to forces in z-matrix, and it
> numerically differentiates the function Z(C) that returns z-matrix
> coordinates, Z, in terms of the cartesians, C, and implements the function
> Z using the python interface to openbabel.

That code would be a good starting point for these helper methods. Would you 
care to share it?

> > > As I may have mentioned, I will probably add support for a number of
> > > other programs, such as GAMESS, NWChem and possibly VASP, at some
> > > point, so obviously should do this in a style that is consistent with
> > > the rest of cclib.
> >
> > We don't parse nwchem or VASP output files at the moment, so there is no
> > parse to currently add code to for those programs.
>
> Well if I were to write some code to parse them, would you want to add it
> to cclib?

Perhaps. To start off with, we could create a branch of cclib that would 
implement just the gradients for VASP, and when we have enough attributes 
parsed compared to other programs, we could include it in the main 
development branch.

Cheers,
Karol

On Friday 12 June 2009 10:41:29 Hugh Chaffey-Millar wrote:
> Hi Karol
>
> > Could I also look over your code and input/output example?
>
> Sure, I'll send these in a separate email to you only, since Noel has it
> already.

Thanks for that code. It's been a while, but I implemented parsing gradients 
today for Gaussian. I write the code from scratch, but used your log files. 
There is a small fragment in the comment of the latest commit. Is that OK?

It's really quite short - that would be revision 850. Here's the diff:
http://cclib.svn.sourceforge.net/viewvc/cclib/trunk/src/cclib/parser/gaussianparser.py?r1=850&r2=849&pathrev=850&diff_format=h

At this point, the parser parses the cartesian forces into a regular mxnx3 
matrix, where n is the number of atoms and m in the number of optimization 
steps.

The numbers are in gaussian-esque hartree/bohr - I guess I should convert that 
to eV/angstrem... what do you say Noel?

Here is one of your log files parsed with the new code:
langner@machine:~$ cat <<EOF | python
> import cclib
> a = cclib.parser.ccopen("MeOMe-grad.log").parse()
> print a.grads
> EOF
[Gaussian MeOMe-grad.log INFO] Creating attribute charge: 0
[Gaussian MeOMe-grad.log INFO] Creating attribute mult: 1
[Gaussian MeOMe-grad.log INFO] Creating attribute atomnos[]
[Gaussian MeOMe-grad.log INFO] Creating attribute natom: 9
[Gaussian MeOMe-grad.log INFO] Creating attribute atomcoords[]
[Gaussian MeOMe-grad.log INFO] Creating attribute nbasis: 57
[Gaussian MeOMe-grad.log INFO] Creating attribute nmo: 57
[Gaussian MeOMe-grad.log INFO] Creating attribute scftargets[]
[Gaussian MeOMe-grad.log INFO] Creating attribute scfenergies[]
[Gaussian MeOMe-grad.log INFO] Creating attribute mosyms[]
[Gaussian MeOMe-grad.log INFO] Creating attribute homos[]
[Gaussian MeOMe-grad.log INFO] Creating attribute moenergies[]
[Gaussian MeOMe-grad.log INFO] Creating attribute grads[]
[Gaussian MeOMe-grad.log INFO] Creating attribute geovalues[]
[Gaussian MeOMe-grad.log INFO] Creating attribute geotargets[]
[Gaussian MeOMe-grad.log INFO] Creating attribute coreelectrons[]
[[[ 0.01026324  0.01389559  0.02445051]
  [ 0.00747792 -0.01446347  0.00280476]
  [ 0.00470098  0.00834956 -0.01982187]
  [-0.0217437  -0.00201811 -0.00282018]
  [-0.00126088  0.00092114 -0.00145291]
  [ 0.00980594 -0.02822511 -0.00185482]
  [-0.02168228  0.00376007  0.00074849]
  [ 0.00764326  0.00369573  0.01417389]
  [ 0.00479553  0.01408459 -0.01622786]]]

It should also work with the current standard geometry optimization tests we 
have, so there's no need for more tests at the moment. I'll add some tests 
soon for the standard tests.

Cheers,
Karol

-- 
written by Karol Langner
Wed Jul 15 17:04:13 CEST 2009