Ticket #632 (closed enhancement: fixed)

Opened 3 years ago

Last modified 3 years ago

pypy compatibility

Reported by: aric Owned by: aric
Priority: normal Milestone: networkx-1.6
Component: networkx Version:
Severity: normal Keywords:
Cc:

Description

Modify code and tests to work with pypy http://pypy.org/

Attachments

benchmark2.py (2.2 KB) - added by aric 3 years ago.
tests_errors_pypy.txt (64.1 KB) - added by jtorrents 3 years ago.

Change History

comment:1 Changed 3 years ago by Aric Hagberg <aric.hagberg@…>

In [3a1af9d7157ddee06cadb7e471e6c9cdecba1ad9/networkx]:

Fixes for pypy compatibility.
Addresses #632

comment:2 Changed 3 years ago by aric

As far as I can tell this updates NetworkX to work with modern versions of pypy. I used the latest nightly build as of today. The parts that use numpy won't work until numpy works fully with pypy.

Most of the changes were to skip the numpy tests since pypy ships with a partial-feature version of numpy. Unfortunately it has the same name so it imports fine and then bombs because it doesn't have most of the functions needed. I added an attribute to the tests (numpy=1) so they can be skipped with

nosetests -A 'not numpy' 

There were a few other interesting bugs that pypy caught, e.g. arbitrary ordering of cyclic permutations in cycle finding needs more careful testing [3a1af9d7157ddee06cadb7e471e6c9cdecba1ad9/networkx#file12]

Don't use "is" or "is not" when you mean "==" or "!=": [3a1af9d7157ddee06cadb7e471e6c9cdecba1ad9/networkx#file10] and [3a1af9d7157ddee06cadb7e471e6c9cdecba1ad9/networkx#file7]

Close files: [3a1af9d7157ddee06cadb7e471e6c9cdecba1ad9/networkx#file16]

This passes all of the tests (except for numpy/scipy, other missing packages etc):

pypy -c "import networkx; networkx.test(numpy=False)"Running NetworkX tests:...............................S
-cut-
----------------------------------------------------------------------
Ran 1096 tests in 19.323s

OK (SKIP=12)

comment:3 Changed 3 years ago by Aric Hagberg <aric.hagberg@…>

In [60275673f52edf9d74f3147b9ce5125a37b96282/networkx]:

Add numpy=True|False flag to networkx.test()
Addresses #632

comment:4 Changed 3 years ago by aric

  • Status changed from new to needinfo

comment:5 Changed 3 years ago by aric

  • Status changed from needinfo to assigned

comment:6 Changed 3 years ago by aric

  • Status changed from assigned to reviewing

comment:7 Changed 3 years ago by aric

Here are some benchmarks on 64bit linux. They are fairly arbitrary but hopefully represent some typical networkx usage. pypy shows pretty big speedup with some methods/algorithms. Surprisingly the degree method and transitivity and clustering algorithms are slower - I didn't look into that. It might be that the benchmark isn't very good?

speedup python2.7 pypy test
4.22 0.5490 0.1301 G.add_nodes_from(range(100)) 
3.67 0.6740 0.1836 G.add_nodes_from(range(100000)) 
4.76 1.3157 0.2765 G.add_nodes_from(nlist); 
5.06 1.5843 0.3131 G.add_edges_from(elist) 
4.95 2.2754 0.4601 G.add_edges_from(elist); 
2.65 2.2606 0.8524 G=nx.fast_gnp_random_graph(10000,0.001); 
1.78 1.6944 0.9504 G=nx.barabasi_albert_graph(10000,3) 
2.93 2.1546 0.7359 sp=nx.shortest_path(G,source=0) 
0.63 0.4368 0.6963 d=G.degree() 
1.87 3.8110 2.0425 m=nx.minimum_spanning_tree(G) 
2.67 2.0439 0.7651 p=nx.pagerank(G) 
2.27 2.7454 1.2094 H=G.copy() 
0.67 6.1665 9.2361 c=nx.clustering(G) 
0.85 2.7344 3.2009 c=nx.transitivity(G) 
1.00 0.7966 0.7988 c=nx.topological_sort(G) 
1.37 2.0273 1.4770 c=list(nx.find_cliques(G)) 

Changed 3 years ago by aric

comment:8 Changed 3 years ago by jtorrents

Aric,

I've run your benchmarks and the results are consistent, oddly transitivity and clustering based on triangles is slower or (for smaller networks) almost the same with pypy and python2.7. I also did some tests with the algorithm for approximation to k-components (#589) and the speedup is quite big (65% faster).

jtorrents@jupiter:~/projects/k_components$ pypy benchmark_kcomponents.py 
Graph used for testing:
Name: gnm_random_graph(200,600)
Type: Graph
Number of nodes: 200
Number of edges: 600
Average degree:   6.0000
3.8064	result=kc.approximation_k_components(G)
0.1079	c=nx.square_clustering(G)
0.0324	c=nx.clustering(G)
0.0329	c=nx.transitivity(G)
jtorrents@jupiter:~/projects/k_components$ python benchmark_kcomponents.py 
Graph used for testing:
Name: gnm_random_graph(200,600)
Type: Graph
Number of nodes: 200
Number of edges: 600
Average degree:   6.0000
11.6567	result=kc.approximation_k_components(G)
0.1801	c=nx.square_clustering(G)
0.0401	c=nx.clustering(G)
0.0366	c=nx.transitivity(G)

comment:9 Changed 3 years ago by jtorrents

I also get an strange error in 2 tests when running NetworkX in pypy (linux 64 bit). I'm not sure of which tests are causing the error. It is difficult to narrow down because if I run nosetests form the command line (ie the nosetests installed in pypy) I get all the ugly errors related to the incomplete version of numpy that comes with pypy. The stack of the errors (same in both errors) is:

======================================================================
ERROR: Failure: AttributeError ('function' object has no attribute 'im_class')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/loader.py", line 495, in makeTest
    return self._makeTest(obj, parent)
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/loader.py", line 544, in _makeTest
    return self.loadTestsFromTestClass(obj)
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/loader.py", line 488, in loadTestsFromTestClass
    for case in filter(wanted, dir(cls))]
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/loader.py", line 486, in wanted
    return sel.wantMethod(item)
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/selector.py", line 175, in wantMethod
    plug_wants = self.plugins.wantMethod(method)
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/plugins/manager.py", line 94, in __call__
    return self.call(*arg, **kw)
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/plugins/manager.py", line 162, in simple
    result = meth(*arg, **kw)
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/plugins/attrib.py", line 287, in wantMethod
    attribs = AttributeGetter(method.im_class, method)
  File "/home/jtorrents/soft/pypy-1.6/site-packages/nose/pyversion.py", line 92, in __getattr__
    return getattr(self._func, attr)
AttributeError: 'function' object has no attribute 'im_class'

I also attach the output of:

pypy -c "import networkx as nx; nx.test(numpy=False, verbosity=2)" &> tests_errors_pypy.txt

Changed 3 years ago by jtorrents

comment:10 Changed 3 years ago by aric

I'm running the same test platform (64 bit linux) without error. So there must be some slightly different setup with nose and pypy.

You can run nosetests from the command line and skip the numpy tests with

nosetests -A 'not numpy' 

comment:11 Changed 3 years ago by jtorrents

I've found one test that generates the error: it is the TestVF2GraphDB in networkx/algorithms/isomorphism/tests/test_isomorphvf2.py. If I comment the decorator @staticmethod the error above is no longer there, but then the test fails because it cannot open the necessary files. I'll try to investigate more and to locate the other test that fails.

comment:12 Changed 3 years ago by jtorrents

The other error is generated also when the @staticmethod decorator is declared in networkx/utils/tests/test_decorators.py. I'm not familiar with decorators and don't have a clue about what is going on. Any ideas or suggestions?

comment:13 Changed 3 years ago by aric

Those work for me. I'm using the dev version of pypy. Maybe that is the difference?

$ pypy -V
Python 2.7.1 (dec57323f88b, Sep 11 2011, 02:02:11)
[PyPy 1.6.0-dev1 with GCC 4.4.3]

comment:14 Changed 3 years ago by jtorrents

I just tested with the nightly build http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 and the error is still there. If I remove the @staticmethod decorator:

class TestVF2GraphDB(object):
    # http://amalfi.dis.unina.it/graph/db/

    @staticmethod
    def create_graph(filename):

to

class TestVF2GraphDB(object):
    # http://amalfi.dis.unina.it/graph/db/

    def create_graph(self,filename):

then no error is raised

comment:15 Changed 3 years ago by jtorrents

I'm using nose version 1.0.0. I get these errors both in my machine at work (Debian stable) and in my laptop (Ubuntu 11.04). Both 64 bit.

comment:16 Changed 3 years ago by aric

I'm using a new version of nose (probably the dev version). It's (I renamed from nosetests to nosetests-pypy)

$nosetests-pypy -V
nosetests-pypy version 1.1.3

Other than that I'm not sure why we see different results.

comment:17 Changed 3 years ago by jtorrents

Yes, that's it! I just installed the development version of nose and the errors are no longer there.

comment:18 Changed 3 years ago by aric

  • Status changed from reviewing to closed
  • Resolution set to fixed

I'm going to close this ticket. The slower performance of the transitivity/clustering code with pypy might be related to the use of set(). I think it is the only algorithm in the benchmarks that uses set in any substantial way.

Note: See TracTickets for help on using tickets.