\chapter{Python Debugging} \label{chap:dbg} One of the most undervalued aspects taught in programming courses is debugging. It's almost as if it's believed that wasting untold hours late at night is good for you! Debugging is an art, but with good principles and good tools, you really can save yourself lots of those late night hours of frustration. \section{The Essence of Debugging: The Principle of Confirmation} Though debugging is an art rather than a science, there are some fundamental principles involved, which will be discussed first. As Pete Salzman and I said in our book on debugging, {\it The Art of Debugging, with GDB, DDD and Eclipse} (No Starch Press, 2008), the following rule is the essence of debugging: \begin{quote} \textbf{The Principle of Confirmation} Fixing a buggy program is a process of confirming, one by one, that the many things you {\it believe} to be true about the code actually {\it are} true. When you find that one of your assumptions is \textit{not} true, you have found a clue to the location (if not the exact nature) of a bug. \end{quote} \noindent Another way of saying this is, \begin{quote} Surprises are good! \end{quote} For example, say you have the following code: \begin{Verbatim}[fontsize=\relsize{-2}] x = y**2 + 3*g(z,2) w = 28 if w+q > 0: u = 1 else: v = 10 \end{Verbatim} Do you think the value of your variable {\bf x} should be 3 after {\bf x} is assigned? Confirm it! Do you think the ``else'' will be executed, not the ``if,'' on that third line? Confirm it! Eventually one of these assertions that you are so sure of will turn out to {\it not} confirm. Then you will have pinpointed the likely {\it location} of the error, thus enabling you to focus on the {\it nature} of the error. \section{Plan for This Chapter} Do NOT debug by simply adding and subtracting {\bf print} statements. Use a debugging tool! If you are not a regular user of a debugging tool, then you are causing yourself unnecessary grief and wasted time; see my debugging slide show, at \url{http://heather.cs.ucdavis.edu/~matloff/debug.html}. The remainder of this chapter will be devoted to various debugging tools. We'll start with Python's basic built-in debugger, PDB. It's pretty primitive, but it's the basis of some other tools, and I'll show you how to exploit its features to make it a pretty decent debugger. Then there will be brief overviews of some other tools. \section{Python's Built-In Debugger, PDB} The built-in debugger for Python, PDB, is rather primitive, but it's very important to understand how it works, for two reasons: \begin{itemize} \item PDB is used indirectly by more sophisticated debugging tools. A good knowledge of PDB will enhance your ability to use those other tools. \item I will show you here how to increase PDB's usefulness even as a standalone debugger. \end{itemize} I will present PDB below in a sequence of increasingly-useful forms: \begin{itemize} \item The basic form. \item The basic form enhanced by strategic use of macros. \item The basic form in conjunction with the Emacs text editor. % \item The basic form in conjunction with the DDD GUI for debuggers. \end{itemize} \subsection{The Basics} Nowadays PDB is an integral part of Python, rather than being invoked as a separate module. % You should be able to find PDB in the {\bf lib} subdirectory of your % Python package. On a Linux system, for example, that is probably in % something like {\bf /usr/lib/python2.2}. {\bf /usr/local/lib/python2.4}, % etc. To debug a script {\bf x.py}, type \begin{Verbatim}[fontsize=\relsize{-2}] % python -m pdb x.py \end{Verbatim} (If {\bf x.py} had had command-line arguments, they would be placed after {\bf x.py} on the command line.) % Of course, since you will use PDB a lot, it would be better % to make an alias for it. For example, under Linux in the C-shell: % % \begin{Verbatim}[fontsize=\relsize{-2}] % alias pdb /usr/lib/python2.6/pdb.py % \end{Verbatim} % % so that you can write more simply % % \begin{Verbatim}[fontsize=\relsize{-2}] % % pdb x.py % \end{Verbatim} Once you are in PDB, set your first breakpoint, say at line 12: \begin{Verbatim}[fontsize=\relsize{-2}] b 12 \end{Verbatim} You can make it conditional, e.g. \begin{Verbatim}[fontsize=\relsize{-2}] b 12, z > 5 \end{Verbatim} Hit {\bf c} (``continue''), which you will get you into {\bf x.py} and then stop at the breakpoint. Then continue as usual, with the main operations being like those of GDB: % \footnote{In most cases, these % commands will not work until you actually get into the debuggee source % code. For example, typing {\bf l} (``list'' lines) prematurely will % produce an error message.} \begin{itemize} \item {\bf b} (``break'') to set a breakpoint \item {\bf tbreak} to set a one-time breakpoint \item {\bf ignore} to specify that a certain breakpoint will be ignored the next k times, where k is specified in the command \item {\bf l} (``list'') to list some lines of source code \item {\bf n} (``next'') to step to the next line, not stopping in function code if the current line is a function call \item {\bf s} (``subroutine'') same as {\bf n}, except that the function {\it is} entered in the case of a call \item {\bf c} (``continue'') to continue until the next break point \item {\bf w} (``where'') to get a stack report \item {\bf u} (``up'') to move up a level in the stack, e.g. to query a local variable there \item {\bf d} (``down'') to move down a level in the stack \item {\bf r} (``return'') continue execution until the current function returns \item {\bf j} (``jump'') to jump to another line {\it without} the intervening code being executed \item {\bf h} (``help'') to get (minimal) online help (e.g. {\bf h b} to get help on the {\bf b} command, and simply {\bf h} to get a list of all commands); type {\bf h pdb} to get a tutorial on PDB\footnote{The tutorial is run through a pager. Hit the space bar to go to the next page, and the q key to quit.} \item {\bf q} (``quit'') to exit PDB \end{itemize} Upon entering PDB, you will get its {\tt (Pdb)} prompt. If you have a multi-file program, breakpoints can be specified in the form {\tt module\_name:line\_number}. For instance, suppose your main module is {\bf x.py}, and it imports {\bf y.py}. You set a breakpoint at line 8 of the latter as follows: \begin{Verbatim}[fontsize=\relsize{-2}] (Pdb) b y:8 \end{Verbatim} Note, though, that you can't do this until {\bf y} has actually been imported by {\bf x}.\footnote{Note also that if the module is implemented in C, you of course will not be able to break there.} When you are running PDB, you are running Python in its interactive mode. Therefore, you can issue any Python command at the PDB prompt. You can set variables, call functions, etc. This can be highly useful. For example, although PDB includes the {\bf p} command for printing out the values of variables and expressions, it usually isn't necessary. To see why, recall that whenever you run Python in interactive mode, simply typing the name of a variable or expression will result in printing it out---exactly what {\bf p} would have done, without typing the `p'. So, if {\bf x.py} contains a variable {\bf ww} and you run PDB, instead of typing \begin{Verbatim}[fontsize=\relsize{-2}] (Pdb) p ww \end{Verbatim} you can simply type \begin{Verbatim}[fontsize=\relsize{-2}] ww \end{Verbatim} and the value of {\bf ww} will be printed to the screen.\footnote{However, if the name of the variable is the same as that of a PDB command (or its abbreviation), the latter will take precedence. If for instance you have a variable {\bf n}, then typing {\bf n} will result in PDB's {\bf n[ext]} command being executed, rather than there being a printout of the value of the variable {\bf n}. To get the latter, you would have to type {\bf p n}.} If your program has a complicated data structure, you could write a function to print to the screen all or part of that structure. Then, since PDB allows you to issue any Python command at the PDB prompt, you could simply call this function at that prompt, thus getting more sophisticated, application-specific printing. After your program either finishes under PDB or runs into an execution error, you can re-run it without exiting PDB---important, since you don't want to lose your breakpoints---by simply hitting {\bf c}. % Note that you % can usually force your program to finish, by the {\bf j} command, % jumping to some statement at the end of the program, and then hitting % {\bf c}. And yes, if you've changed your source code since then, the change will be reflected in PDB.\footnote{PDB is, as seen above, just a Python program itself. When you restart, it will re-import your source code. By the way, the reason your breakpoints are retained is that of course they are variables in PDB. Specifically, they are stored in member variable named {\bf breaks} in the the {\bf Pdb} class in {\bf pdb.py}. That variable is set up as a dictionary, with the keys being names of your {\bf .py} source files, and the items being the lists of breakpoints.} If you give PDB a single-step command like {\bf n} when you are on a Python line which does multiple operations, you will need to issue the {\bf n} command multiple times (or set a temporary breakpoint to skip over this). For example, \begin{Verbatim}[fontsize=\relsize{-2}] for i in range(10): \end{Verbatim} does two operations. It first calls {\bf range()}, and then sets {\bf i}, so you would have to issue {\bf n} twice. And how about this one? \begin{Verbatim}[fontsize=\relsize{-2}] y = [(y,x) for (x,y) in x] \end{Verbatim} If {\bf x} has, say, 10 elements, then you would have to issue the {\tt n} command 10 times! Here you would definitely want to set a temporary breakpoint to get around it. \subsection{Using PDB Macros} PDB's undeniably bare-bones nature can be remedied quite a bit by making good use of the {\bf alias} command, which I strongly suggest. For example, type \begin{Verbatim}[fontsize=\relsize{-2}] alias c c;;l \end{Verbatim} This means that each time you hit {\bf c} to continue, when you next stop at a breakpoint you automatically get a listing of the neighboring code. This will really do a lot to make up for PDB's lack of a GUI. In fact, this is so important that you should put it in your PDB startup file, which in Linux is {\bf \$HOME/.pdbrc}.\footnote{Python will also check for such a file in your current directory.} That way the alias is always available. You could do the same for the {\bf n} and {\bf s} commands: \begin{Verbatim}[fontsize=\relsize{-2}] alias c c;;l alias n n;;l alias s s;;l \end{Verbatim} There is an {\bf unalias} command too, to cancel an alias. You can write other macros which are specific to the particular program you are debugging. For example, let's again suppose you have a variable named {\bf ww} in {\bf x.py}, and you wish to check its value each time the debugger pauses, say at breakpoints. Then change the above alias to \begin{Verbatim}[fontsize=\relsize{-2}] alias c c;;l;;ww \end{Verbatim} \subsection{Using \_\_dict\_\_} \label{pdbdict} In Section \ref{classdict} below, we'll show that if {\bf o} is an object of some class, then printing {\bf o.\_\_dict\_\_} will print all the member variables of this object. Again, you could combine this with PDB's alias capability, e.g. \begin{Verbatim}[fontsize=\relsize{-2}] alias c c;;l;;o.__dict__ \end{Verbatim} Actually, it would be simpler and more general to use \begin{Verbatim}[fontsize=\relsize{-2}] alias c c;;l;;self \end{Verbatim} This way you get information on the member variables no matter what class you are in. On the other hand, this apparently does not produce information on member variables in the parent class. \subsection{The type() Function} In reading someone else's code, or even one's own, one might not be clear what type of object a variable currently references. For this, the {\bf type()} function is sometimes handy. Here are some examples of its use: \begin{Verbatim}[fontsize=\relsize{-2}] >>> x = [5,12,13] >>> type(x) >>> type(3) >>> def f(y): return y*y ... >>> f(5) 25 >>> type(f) \end{Verbatim} \subsection{Using PDB with Emacs} Emacs is a combination text editor and tools collection. Many software engineers swear by it. It is available for Windows, Macs and Linux. But even if you are not an Emacs aficionado, you may find it to be an excellent way to use PDB. You can split Emacs into two windows, one for editing your program and the other for PDB. As you step through your code in the second window, you can see yourself progress through the code in the first. To get started, say on your file {\bf x.py}, go to a command window (whatever you have under your operating system), and type either \begin{Verbatim}[fontsize=\relsize{-2}] emacs x.py \end{Verbatim} or \begin{Verbatim}[fontsize=\relsize{-2}] emacs -nw x.py \end{Verbatim} The former will create a new Emacs window, where you will have mouse operations available, while the latter will run Emacs in text-only operations in the current window. I'll call the former ``GUI mode.'' Then type {\bf M-x pdb}, where for most systems ``M,'' which stands for ``meta,'' means the Escape (or Alt) key rather than the letter M. You'll be asked how to run PDB; answer in the manner you would run PDB externally to Emacs (but with a full path name), including arguments, e.g. \begin{Verbatim}[fontsize=\relsize{-2}] /usr/local/lib/python2.4/pdb.py x.py 3 8 \end{Verbatim} where the 3 and 8 in this example are your program's command-line arguments. At that point Emacs will split into two windows, as described earlier. You can set breakpoints directly in the PDB window as usual, or by hitting {\bf C-x space} at the desired line in your program's window; here and below, ``C-'' means hitting the control key and holding it while you type the next key. At that point, run PDB as usual. If you change your program and are using the GUI version of Emacs, hit IM-Python $|$ Rescan to make the new version of your program known to PDB. In addition to coordinating PDB with your error, note that another advantage of Emacs in this context is that Emacs will be in Python mode, which gives you some extra editing commands specific to Python. I'll describe them below. In terms of general editing commands, plug ``Emacs tutorial'' or ``Emacs commands'' into your favorite Web search engine, and you'll see tons of resources. Here I'll give you just enough to get started. First, there is the notion of a {\bf buffer}. Each file you are editing\footnote{There may be several at once, e.g. if your program consists of two or more source files.} has its own buffer. Each other action you take produces a buffer too. For instance, if you invoke one of Emacs' online help commands, a buffer is created for it (which you can edit, save, etc. if you wish). An example relevant here is PDB. When you do {\bf M-x pdb}, that produces a buffer for it. So, at any given time, you may have several buffers. You also may have several windows, though for simplicity we'll assume just two windows here. In the following table, we show commands for both the text-only and the GUI versions of Emacs. Of course, you can use the text-based commands in the GUI too. \begin{tabular}{|l|l|l|} \hline action & text & GUI \\ \hline \hline cursor movement & arrow keys, PageUp/Down & mouse, left scrollbar \\ \hline undo & C-x u & Edit $|$ Undo \\ \hline cut & C-space (cursor move) C-w & select region $|$ Edit $|$ Cut \\ \hline paste & C-y & Edit $|$ Paste \\ \hline search for string & C-s & Edit $|$ Search \\ \hline mark region & C-@ & select region \\ \hline go to other window & C-x o & click window \\ \hline enlarge window & (1 line at a time) C-x \^ & drag bar \\ \hline repeat folowing command n times & M-x n & M-x n \\ \hline list buffers & C-x C-b & Buffers \\ \hline go to a buffer & C-x b & Buffers \\ \hline exit Emacs & C-x C-c & File $|$ Exit Emacs \\ \hline \end{tabular} In using PDB, keep in mind that the name of your PDB buffer will begin with ``gud,'' e.g. {\bf gud-x.py}. You can get a list of special Python operations in Emacs by typing {\bf C-h d} and then requesting info in {\bf python-mode}. One nice thing right off the bat is that Emacs' {\bf python-mode} adds a special touch to auto-indenting: It will automatically indent further right after a {\bf def} or {\bf class} line. Here are some operations: \begin{tabular}{|l|l|l|} action & text & GUI \\ \hline \hline comment-out region & C-space (cursor move) C-c \# & select region $|$ Python $|$ Comment \\ \hline go to start of def or class & ESC C-a & ESC C-a \\ \hline go to end of def or class & ESC C-e & ESC C-e \\ \hline go one block outward & C-c C-u & C-c C-u \\ \hline shift region right & mark region, C-c C-r & mark region, Python $|$ Shift right \\ \hline shift region left & mark region, C-c C-l & mark region, Python $|$ Shift left \\ \hline \end{tabular} % \subsection{Debugging with DDD} % % DDD, available on many Linux systems (and freely downloadable if your % system doesn't have it), is a GUI for many debuggers, such GDB (for % C/C++), JDB (for Java), Perl's built-in Perl debugger, and so on. It % can be used on PDB for Python, and thus make your usage of PDB more % enjoyable and productive. % % \subsubsection{You'll Need a Special PYDB} % % The first use of DDD on PDB was designed by Richard Wolff. He modified % {\bf pdb.py} slightly for this purpose, calling the new PDB {\bf % pydb.py}. It was designed for Python 1.5, but he has kindly provided an % update for me. You'll need the files % % \begin{Verbatim}[fontsize=\relsize{-2}] % http://heather.cs.ucdavis.edu/~matloff/Python/DDD/pydb.py % http://heather.cs.ucdavis.edu/~matloff/Python/DDD/pydbcmd.py % http://heather.cs.ucdavis.edu/~matloff/Python/DDD/pydbsupt.py % \end{Verbatim} % % Place the files somewhere in your search path, say {\bf /usr/bin}. % Make sure that you give them execute permission, and that {\bf bdb.py} % from the Python library is in your {\bf PYTHONPATH}. % % \subsubsection{DDD Launch and Program Loading} % % To start, say for debugging {\bf fme2.py} in Section \ref{namemain}, % first make sure that {\bf main()} is set up as described in that % section. % % When you invoke DDD, tell it to use PYDB: % % \begin{Verbatim}[fontsize=\relsize{-2}] % ddd --debugger /usr/bin/pydb.py % \end{Verbatim} % % Then in DDD's Console, i.e. the PDB command subwindow (near the bottom), % type % % \begin{Verbatim}[fontsize=\relsize{-2}] % (pdb) file fme2.py % \end{Verbatim} % % Later, when you make a change to your source code, again issue the command % % \begin{Verbatim}[fontsize=\relsize{-2}] % (pdb) file fme2.py % \end{Verbatim} % % Your breakpoints from the last run will be retained. % % Select Program $|$ Run as usual to set your command-line arguments, and % then run. (You may get a ``DDD: No Source'') popup error window, but % just click OK and ignore it.) % % \subsubsection{Breakpoints} % % To set a breakpoint, right-click somewhere in blank space on the line in % your source file and choose Set Breakpoint (or Set Temporary Breakpoint % or Continue to Until Here, as the case may be). % % \subsubsection{Running Your Code} % % To run, click on Program $|$ Run, fill in your program's command line % arguments if any in the Run with Arguments box, and click Run in that % pop-up window. You will be asked to hit Continue, which you could do by % clicking Program $|$ Run $|$, but is more conveniently done by clicking % Continue in the little command summary window. (But don't use Run % there.) % % You can then click on Next, Step, Cont etc. The marker for the current % execution line is shaped like an `I', though rather faint when the mouse % pointer is not in the source code section of the DDD window.. % % By the way, do not refer to {\bf sys.argv} in freestanding code within a % class. When your program is first loaded, any freestanding code will be % executed, and since the command-line arguments won't have been loaded % yet, so you will get an ``index out of range'' error. Avoid this by % putting code involving {\bf sys.argv} either inside a function in the % class, or outside the class entirely. % % \subsubsection{Inspecting Variables} % % You can inspect the value of a variable by moving the mouse pointer to % any instance of the variable in the source code window. % % As mentioned in Section \ref{pdbdict}, if {\bf o} is an object of some % class, then printing {\bf o.\_\_dict\_\_} will print all the member % variables of this object. In DDD, you can do this even more easily, as % follows. Simply put that expression in a comment, e.g. % % \begin{Verbatim}[fontsize=\relsize{-2}] % # o.__dict__ % \end{Verbatim} % % and then whenever you wish to inspect the member variables of {\bf o}, % simply move the mouse pointer to that expression in the comment! % % Make good use of DDD's feature which allows a variable to be displayed % continuously. Simply right-click on any instance of the variable, and % then choose Display. % % \subsubsection{Miscellaneous} % % DDD, developed originally for C/C++, is not always a perfect match to % Python. But since what DDD actually does is relay your clicked commands to % PDB, as you can see in DDD's Console, whatever DDD can't do for you, % you can type PDB commands directly into the Console. \section{Debugging with Xpdb} Well, this one is my own creation. I developed it under the premise that PDB would be fine if only it had a window in which to watch my movement through my source code (as with Emacs above). Try it! Very easy to set up and use. Go to \url{http://heather.cs.ucdavis.edu/~matloff/Python/Xpdb}. \section{Debugging with Winpdb (GUI)} The Winpdb debugger (\url{http://winpdb.org/}),\footnote{No, it's not just for Microsoft Windows machines, in spite of the name.} is very good. Its functionality is excellent, and its GUI is very attractive visually. Among other things, it can be used to debug threaded code, curses-based code and so on, which many debuggers can't. Winpdb is a GUI front end to the text-based RPDB2, which is in the same package. I have a tutorial on both at \url{http://heather.cs.ucdavis.edu/~matloff/winpdb.html}. \section{Debugging with Eclipse (GUI)} I personally do not like integrated development environments (IDEs). They tend to be very slow to load, often do not allow me to use my favorite text editor,\footnote{I use vim, but the main point is that I want to use the same editor for all my work---programming, writing, e-mail, Web site development, etc.} and in my view they do not add much functionality. However, if you are a fan of IDEs, here are some suggestions: However, if you like IDEs, I do suggest Eclipse, which I have a tutorial for at \url{http://heather.cs.ucdavis.edu/~matloff/eclipse.html}. My tutorial is more complete than most, enabling you to avoid the ``gotchas'' and have smooth sailing. \section{Debugging with PUDB} What a nice little tool! Uses Curses, so its screen footprint is tiny (same as your terminal, as it runs there). Use keys like n for Next, as usual. Variables, stack etc. displayed in right-hand half of the screen. See \url{http://heather.cs.ucdavis.edu/~matloff/pudb.htm} for a slideshow-like tutorial. \section{Some Python Internal Debugging Aids} There are various built-in functions in Python that you may find helpful during the debugging process. \subsection{The {\tt str()} Method} The built-in method {\bf str()} converts objects to strings. For scalars or lists, this has the obvious effect, e.g. \begin{Verbatim}[fontsize=\relsize{-2}] >>> x = [1,2,3] >>> str(x) '[1, 2, 3]' \end{Verbatim} But what if {\bf str()} is applied to an instance of a class? If for example we run our example program from Section \ref{tfe} in PDB, we would get a result like this \begin{Verbatim}[fontsize=\relsize{-2}] (Pdb) str(b) '' \end{Verbatim} This might not be too helpful. However, we can override {\bf str()} as follows. Within the definition of the class {\bf textfile}, we can override the built-in method {\bf \_\_str\_\_()}, which is the method that defines the effect of applying {\bf str()} to objects of this class. We add the following code to the definition of the class {\bf textfile}: \begin{Verbatim}[fontsize=\relsize{-2}] def __str__(self): return self.name+' '+str(self.nlines)+' '+str(self.nwords) \end{Verbatim} (Note that the method is required to return a string.) Running in PDB now, we get \begin{Verbatim}[fontsize=\relsize{-2}] (Pdb) str(b) 'y 3 3' \end{Verbatim} Again, you could arrange so that were printed automatically, at every pause, e.g. \begin{Verbatim}[fontsize=\relsize{-2}] alias c c;;l;;str(b) \end{Verbatim} \subsection{The {\tt locals()} Function} \label{locals} The {\bf locals()} function, for instance, will print out all the local variables in the current item (class instance, method, etc.). For example, in the code {\bf tfe.py} from Section \ref{tfe}, let's put breakpoints at the lines \begin{Verbatim}[fontsize=\relsize{-2}] self.nwords += len(w) \end{Verbatim} and \begin{Verbatim}[fontsize=\relsize{-2}] print "the number of text files open is", textfile.ntfiles \end{Verbatim} and then call {\bf locals()} from the PDB command line the second and first times we hit those breakpoints, respectively. Here is what we get: \begin{Verbatim}[fontsize=\relsize{-2}] (Pdb) locals() {'self': <__main__.textfile instance at 0x8187c0c>, 'l': 'Here is an example\n', 'w': ['Here', 'is', 'an', 'example']} ... (Pdb) c > /www/matloff/public_html/Python/tfe.py(27)main() -> print "the number of text files open is", textfile.ntfiles (Pdb) locals() {'a': <__main__.textfile instance at 0x8187c0c>, 'b': <__main__.textfile instance at 0x81c0654>} \end{Verbatim} There is also a built-in {\bf globals()} function which plays a similar role, \subsection{The {\tt \_\_dict\_\_} Attribute} \label{classdict} This was covered in Section \ref{pdbdict}. \subsection{The {\tt id()} Function} Sometimes it is helpful to know the actual memory address of an object. For example, you may have two variables which you think point to the same object, but are not sure. The {\bf id()} method will give you the address of the object. For example: \begin{Verbatim}[fontsize=\relsize{-2}] >>> x = [1,2,3] >>> id(x) -1084935956 >>> id(x[1]) 137809316 \end{Verbatim} (Don't worry about the ``negative'' address, which just reflects the fact that the address was so high that, viewed as a 2s-complement integer, it is ``negative.'')