This article explains the new features in Python 2.3. Python 2.3 was released on July 29, 2003.

The main themes for Python 2.3 are polishing some of the features added in 2.2, adding various small but useful enhancements to the core language, and expanding the standard library. The new object model introduced in the previous version has benefited from 18 months of bugfixes and from optimization efforts that have improved the performance of new-style classes. A few new built-in functions have been added such as sum() and enumerate(). The in operator can now be used for substring searches (e.g. "ab" in "abc" returns True).

Some of the many new library features include Boolean, set, heap, and date/time data types, the ability to import modules from ZIP-format archives, metadata support for the long-awaited Python catalog, an updated version of IDLE, and modules for logging messages, wrapping text, parsing CSV files, processing command-line options, using BerkeleyDB databases… the list of new and enhanced modules is lengthy.

This article doesn’t attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.3, such as the Python Library Reference and the Python Reference Manual. If you want to understand the complete implementation and design rationale, refer to the PEP for a particular new feature.

Original source

PEP 218: Built-In Set Objects

Python 2.3 introduced the sets module. C implementations of set data types have now been added to the Python core as two new built-in types, set(iterable) and frozenset(iterable). They provide high speed operations for membership testing, for eliminating duplicates from sequences, and for mathematical operations like unions, intersections, differences, and symmetric differences.

The frozenset() type is an immutable version of set(). Since it is immutable and hashable, it may be used as a dictionary key or as a member of another set.

The sets module remains in the standard library, and may be useful if you wish to subclass the Set or ImmutableSet classes. There are currently no plans to deprecate the module.

PEP 237: Unifying Long Integers and Integers

The lengthy transition process for this PEP, begun in Python 2.2, takes another step forward in Python 2.4. In 2.3, certain integer operations that would behave differently after int/long unification triggered FutureWarning warnings and returned values limited to 32 or 64 bits (depending on your platform). In 2.4, these expressions no longer produce a warning and instead produce a different result that’s usually a long integer.

The problematic expressions are primarily left shifts and lengthy hexadecimal and octal constants. For example, 2 << 32 results in a warning in 2.3, evaluating to 0 on 32-bit platforms. In Python 2.4, this expression now returns the correct answer, 8589934592.

PEP 289: Generator Expressions

The iterator feature introduced in Python 2.2 and the itertools module make it easier to write programs that loop through large data sets without having the entire data set in memory at one time. List comprehensions don’t fit into this picture very well because they produce a Python list object containing all of the items. This unavoidably pulls all of the objects into memory, which can be a problem if your data set is very large. When trying to write a functionally styled program, it would be natural to write something like:

instead of

The first form is more concise and perhaps more readable, but if you’re dealing with a large number of link objects you’d have to write the second form to avoid having all link objects in memory at the same time.

Generator expressions work similarly to list comprehensions but don’t materialize the entire list; instead they create a generator that will return elements one by one. The above example could be written as:

Generator expressions always have to be written inside parentheses, as in the above example. The parentheses signalling a function call also count, so if you want to create an iterator that will be immediately passed to a function you could write:

Generator expressions differ from list comprehensions in various small ways. Most notably, the loop variable (obj in the above example) is not accessible outside of the generator expression. List comprehensions leave the variable assigned to its last value; future versions of Python will change this, making list comprehensions match generator expressions in this respect.

PEP 292: Simpler String Substitutions

Some new classes in the standard library provide an alternative mechanism for substituting variables into strings; this style of substitution may be better for applications where untrained users need to edit templates.

The usual way of substituting variables by name is the % operator:

When writing the template string, it can be easy to forget the i or s after the closing parenthesis. This isn’t a big problem if the template is in a Python module, because you run the code, get an “Unsupported format character” ValueError, and fix the problem. However, consider an application such as Mailman where template strings or translations are being edited by users who aren’t aware of the Python language. The format string’s syntax is complicated to explain to such users, and if they make a mistake, it’s difficult to provide helpful feedback to them.

PEP 292 adds a Template class to the string module that uses $ to indicate a substitution:

If a key is missing from the dictionary, the substitute() method will raise a KeyError. There’s also a safe_substitute() method that ignores missing keys:

PEP 318: Decorators for Functions and Methods

Python 2.2 extended Python’s object model by adding static methods and class methods, but it didn’t extend Python’s syntax to provide any new way of defining static or class methods. Instead, you had to write a def statement in the usual way, and pass the resulting method to a staticmethod() or classmethod() function that would wrap up the function as a method of the new type. Your code would look like this:

If the method was very long, it would be easy to miss or forget the classmethod() invocation after the function body.

The intention was always to add some syntax to make such definitions more readable, but at the time of 2.2’s release a good syntax was not obvious. Today a good syntax still isn’t obvious but users are asking for easier access to the feature; a new syntactic feature has been added to meet this need.

The new feature is called “function decorators”. The name comes from the idea that classmethod(), staticmethod(), and friends are storing additional information on a function object; they’re decorating functions with more details.

The notation borrows from Java and uses the @ character as an indicator. Using the new syntax, the example above would be written:

The @classmethod is shorthand for the meth=classmethod(meth) assignment. More generally, if you have the following:

It’s equivalent to the following pre-decorator code:

Decorators must come on the line before a function definition, one decorator per line, and can’t be on the same line as the def statement, meaning that @A def f(): ... is illegal. You can only decorate function definitions, either at the module level or inside a class; you can’t decorate class definitions.

A decorator is just a function that takes the function to be decorated as an argument and returns either the same function or some new object. The return value of the decorator need not be callable (though it typically is), unless further decorators will be applied to the result. It’s easy to write your own decorators. The following simple example just sets an attribute on the function object:

As a slightly more realistic example, the following decorator checks that the supplied argument is an integer:

An example in PEP 318 contains a fancier version of this idea that lets you both specify the required type and check the returned type.

Decorator functions can take arguments. If arguments are supplied, your decorator function is called with only those arguments and must return a new decorator function; this function must take a single function and return a function, as previously described. In other words, @A @B @C(args) becomes:

Getting this right can be slightly brain-bending, but it’s not too difficult.

A small related change makes the func_name attribute of functions writable. This attribute is used to display function names in tracebacks, so decorators should change the name of any new function that’s constructed and returned.

PEP 322: Reverse Iteration

A new built-in function, reversed(seq), takes a sequence and returns an iterator that loops over the elements of the sequence in reverse order.

Compared to extended slicing, such as range(1,4)[::-1], reversed() is easier to read, runs faster, and uses substantially less memory.

Note that reversed() only accepts sequences, not arbitrary iterators. If you want to reverse an iterator, first convert it to a list with list().

PEP 324: New subprocess Module

The standard library provides a number of ways to execute a subprocess, offering different features and different levels of complexity.

os.system(command) is easy to use, but slow (it runs a shell process which executes the command) and dangerous (you have to be careful about escaping the shell’s metacharacters). The popen2 module offers classes that can capture standard output and standard error from the subprocess, but the naming is confusing. The subprocess module cleans this up, providing a unified interface that offers all the features you might need.

Instead of popen2’s collection of classes, subprocess contains a single class called subprocess.Popen whose constructor supports a number of different keyword arguments.

args is commonly a sequence of strings that will be the arguments to the program executed as the subprocess. (If the shell argument is true, args can be a string which will then be passed on to the shell for interpretation, just as os.system() does.)

stdin, stdout, and stderr specify what the subprocess’s input, output, and error streams will be. You can provide a file object or a file descriptor, or you can use the constant subprocess.PIPE to create a pipe between the subprocess and the parent.

The constructor has a number of handy options:

Once you’ve created the Popen instance, you can call its wait() method to pause until the subprocess has exited, poll() to check if it’s exited without pausing, or communicate(data) to send the string data to the subprocess’s standard input.

communicate(data) then reads any data that the subprocess has sent to its standard output or standard error, returning a tuple (stdout_data, stderr_data).

call() is a shortcut that passes its arguments along to the Popen constructor, waits for the command to complete, and returns the status code of the subprocess. It can serve as a safer analog to os.system():

The command is invoked without use of the shell. If you really do want to use the shell, you can add shell=True as a keyword argument and provide a string instead of a sequence:

The PEP takes various examples of shell and Python code and shows how they’d be translated into Python code that uses subprocess. Reading this section of the PEP is highly recommended.

PEP 327: Decimal Data Type

Python has always supported floating-point (FP) numbers, based on the underlying C double type, as a data type. However, while most programming languages provide a floating-point type, many people (even programmers) are unaware that floating-point numbers don’t represent certain decimal fractions accurately. The new Decimal type can represent these fractions accurately, up to a user-specified precision limit.

Why is Decimal needed?

The limitations arise from the representation used for floating-point numbers. FP numbers are made up of three components:

• The sign, which is positive or negative.
• The mantissa, which is a single-digit binary number followed by a fractional part. For example, 1.01 in base-2 notation is 1 + 0/2 + 1/4, or 1.25 in decimal notation.
• The exponent, which tells where the decimal point is located in the number represented.

For example, the number 1.25 has positive sign, a mantissa value of 1.01 (in binary), and an exponent of 0 (the decimal point doesn’t need to be shifted). The number 5 has the same sign and mantissa, but the exponent is 2 because the mantissa is multiplied by 4 (2 to the power of the exponent 2); 1.25 * 4 equals 5.

Modern systems usually provide floating-point support that conforms to a standard called IEEE 754. C’s double type is usually implemented as a 64-bit IEEE 754 number, which uses 52 bits of space for the mantissa. This means that numbers can only be specified to 52 bits of precision. If you’re trying to represent numbers whose expansion repeats endlessly, the expansion is cut off after 52 bits. Unfortunately, most software needs to produce output in base 10, and common fractions in base 10 are often repeating decimals in binary. For example, 1.1 decimal is binary 1.0001100110011 ... ; .1 = 1/16 + 1/32 + 1/256 plus an infinite number of additional terms. IEEE 754 has to chop off that infinitely repeated decimal after 52 digits, so the representation is slightly inaccurate.

Sometimes you can see this inaccuracy when the number is printed:

The inaccuracy isn’t always visible when you print the number because the FP-to-decimal-string conversion is provided by the C library, and most C libraries try to produce sensible output. Even if it’s not displayed, however, the inaccuracy is still there and subsequent operations can magnify the error.

For many applications this doesn’t matter. If I’m plotting points and displaying them on my monitor, the difference between 1.1 and 1.1000000000000001 is too small to be visible. Reports often limit output to a certain number of decimal places, and if you round the number to two or three or even eight decimal places, the error is never apparent. However, for applications where it does matter, it’s a lot of work to implement your own custom arithmetic routines.

Hence, the Decimal type was created.

The Decimal type

A new module, decimal, was added to Python’s standard library. It contains two classes, Decimal and Context.

Decimal instances represent numbers, and Context instances are used to wrap up various settings such as the precision and default rounding mode.

Decimal instances are immutable, like regular Python integers and FP numbers; once it’s been created, you can’t change the value an instance represents.

Decimal instances can be created from integers or strings:

You can also provide tuples containing the sign, the mantissa represented as a tuple of decimal digits, and the exponent:

Cautionary note: the sign bit is a Boolean value, so 0 is positive and 1 is negative.

Converting from floating-point numbers poses a bit of a problem: should the FP number representing 1.1 turn into the decimal number for exactly 1.1, or for 1.1 plus whatever inaccuracies are introduced? The decision was to dodge the issue and leave such a conversion out of the API. Instead, you should convert the floating-point number into a string using the desired precision and pass the string to the Decimal constructor:

Once you have Decimal instances, you can perform the usual mathematical operations on them. One limitation: exponentiation requires an integer exponent:

You can combine Decimal instances with integers, but not with floating-point numbers:

Decimal numbers can be used with the math and cmath modules, but note that they’ll be immediately converted to floating-point numbers before the operation is performed, resulting in a possible loss of precision and accuracy. You’ll also get back a regular floating-point number and not a Decimal.

Decimal instances have a sqrt() method that returns a Decimal, but if you need other things such as trigonometric functions you’ll have to implement them.

The Context type

Instances of the Context class encapsulate several settings for decimal operations:

There’s a thread-local default context available by calling getcontext(); you can change the properties of this context to alter the default precision, rounding, or trap handling. The following example shows the effect of changing the precision of the default context:

The default action for error conditions is selectable; the module can either return a special value such as infinity or not-a-number, or exceptions can be raised:

The Context instance also has various methods for formatting numbers such as to_eng_string() and to_sci_string().

For more information, see the documentation for the decimal module, which includes a quick-start tutorial and a reference.

PEP 328: Multi-line Imports

One language change is a small syntactic tweak aimed at making it easier to import many names from a module. In a from module import names statement, names is a sequence of names separated by commas. If the sequence is very long, you can either write multiple imports from the same module, or you can use backslashes to escape the line endings like this:

The syntactic change in Python 2.4 simply allows putting the names within parentheses. Python ignores newlines within a parenthesized expression, so the backslashes are no longer needed:

The PEP also proposes that all import statements be absolute imports, with a leading . character to indicate a relative import. This part of the PEP was not implemented for Python 2.4, but was completed for Python 2.5.

PEP 331: Locale-Independent Float/String Conversions

The locale modules lets Python software select various conversions and display conventions that are localized to a particular country or language. However, the module was careful to not change the numeric locale because various functions in Python’s implementation required that the numeric locale remain set to the 'C' locale. Often this was because the code was using the C library’s atof() function.

Not setting the numeric locale caused trouble for extensions that used third-party C libraries, however, because they wouldn’t have the correct locale set. The motivating example was GTK+, whose user interface widgets weren’t displaying numbers in the current locale.

The solution described in the PEP is to add three new functions to the Python API that perform ASCII-only conversions, ignoring the locale setting:

The code for these functions came from the GLib library (), whose developers kindly relicensed the relevant functions and donated them to the Python Software Foundation. The locale module can now change the numeric locale, letting extensions such as GTK+ produce the correct results.

Other Language Changes

Here are all of the changes that Python 2.4 makes to the core Python language.

• Decorators for functions and methods were added (PEP 318).
• Built-in set() and frozenset() types were added (PEP 218). Other new built-ins include the reversed(seq) function (PEP 322).
• Generator expressions were added (PEP 289).
• Certain numeric expressions no longer return values restricted to 32 or 64 bits (PEP 237).
• You can now put parentheses around the list of names in a from module import names statement (PEP 328).
• The dict.update() method now accepts the same argument forms as the dict constructor. This includes any mapping, any iterable of key/value pairs, and keyword arguments. (Contributed by Raymond Hettinger.)
• The string methods ljust(), rjust(), and center() now take an optional argument for specifying a fill character other than a space. (Contributed by Raymond Hettinger.)
• Strings also gained an rsplit() method that works like the split() method but splits from the end of the string. (Contributed by Sean Reifschneider.)
• Three keyword parameters, cmp, key, and reverse, were added to the sort() method of lists. These parameters make some common usages of sort() simpler. All of these parameters are optional.
• There is a new built-in function sorted(iterable) that works like the in-place list.sort() method but can be used in expressions. The differences are:
• Integer operations will no longer trigger an OverflowWarning. The OverflowWarning warning will disappear in Python 2.5.
• The interpreter gained a new switch, -m, that takes a name, searches for the corresponding module on sys.path, and runs the module as a script. For example, you can now run the Python profiler with python -m profile. (Contributed by Nick Coghlan.)
• The eval(expr, globals, locals) and execfile(filename, globals, locals) functions and the exec statement now accept any mapping type for the locals parameter. Previously this had to be a regular Python dictionary. (Contributed by Raymond Hettinger.)
• The zip() built-in function and itertools.izip() now return an empty list if called with no arguments. Previously they raised a TypeError exception. This makes them more suitable for use with variable length argument lists:
• Encountering a failure while importing a module no longer leaves a partially initialized module object in sys.modules. The incomplete module object left behind would fool further imports of the same module into succeeding, leading to confusing errors. (Fixed by Tim Peters.)
• None is now a constant; code that binds a new value to the name None is now a syntax error. (Contributed by Raymond Hettinger.)

Optimizations

The inner loops for list and tuple slicing were optimized and now run about one-third faster. The inner loops for dictionaries were also optimized, resulting in performance boosts for keys(), values(), items(), iterkeys(), itervalues(), and iteritems(). (Contributed by Raymond Hettinger.)

The machinery for growing and shrinking lists was optimized for speed and for space efficiency. Appending and popping from lists now runs faster due to more efficient code paths and less frequent use of the underlying system realloc(). List comprehensions also benefit.

list.extend() was also optimized and no longer converts its argument into a temporary list before extending the base list. (Contributed by Raymond Hettinger.)

list(), tuple(), map(), filter(), and zip() now run several times faster with non-sequence arguments that supply a len() method. (Contributed by Raymond Hettinger.)

The methods list.getitem(), dict.getitem(), and dict.contains() are now implemented as method_descriptor objects rather than wrapper_descriptor objects. This form of access doubles their performance and makes them more suitable for use as arguments to functionals: map(mydict.getitem, keylist).

Added a new opcode, LIST_APPEND, that simplifies the generated bytecode for list comprehensions and speeds them up by about a third. (Contributed by Raymond Hettinger.)

The peephole bytecode optimizer has been improved to produce shorter, faster bytecode; remarkably, the resulting bytecode is more readable. (Enhanced by Raymond Hettinger.)

String concatenations in statements of the form s = s + "abc" and s += "abc" are now performed more efficiently in certain circumstances. This optimization won’t be present in other Python implementations such as Jython, so you shouldn’t rely on it; using the join() method of strings is still recommended when you want to efficiently glue a large number of strings together. (Contributed by Armin Rigo.)

The net result of the 2.4 optimizations is that Python 2.4 runs the pystone benchmark around 5% faster than Python 2.3 and 35% faster than Python 2.2. (pystone is not a particularly good benchmark, but it’s the most commonly used measurement of Python’s performance. Your own applications may show greater or smaller benefits from Python 2.4.)

New, Improved, and Deprecated Modules

As usual, Python’s standard library received a number of enhancements and bug fixes. Here’s a partial list of the most notable changes, sorted alphabetically by module name. Consult the Misc/NEWS file in the source tree for a more complete list of changes, or look through the CVS logs for all the details.

• The asyncore module’s loop() function now has a count parameter that lets you perform a limited number of passes through the polling loop. The default is still to loop forever.

• The base64 module now has more complete RFC 3548 support for Base64, Base32, and Base16 encoding and decoding, including optional case folding and optional alternative alphabets. (Contributed by Barry Warsaw.)

• The bisect module now has an underlying C implementation for improved performance. (Contributed by Dmitry Vasiliev.)

• The CJKCodecs collections of East Asian codecs, maintained by Hye-Shik Chang, was integrated into 2.4. The new encodings are:

  • Chinese (PRC): gb2312, gbk, gb18030, big5hkscs, hz
  • Chinese (ROC): big5, cp950
  • Japanese: cp932, euc-jis-2004, euc-jp, euc-jisx0213, iso-2022-jp, iso-2022-jp-1, iso-2022-jp-2, iso-2022-jp-3, iso-2022-jp-ext, iso-2022-jp-2004, shift-jis, shift-jisx0213, shift-jis-2004
  • Korean: cp949, euc-kr, johab, iso-2022-kr

• Some other new encodings were added: HP Roman8, ISO_8859-11, ISO_8859-16, PCTP-154, and TIS-620.

• The UTF-8 and UTF-16 codecs now cope better with receiving partial input. Previously the StreamReader class would try to read more data, making it impossible to resume decoding from the stream. The read() method will now return as much data as it can and future calls will resume decoding where previous ones left off. (Implemented by Walter Dörwald.)

• There is a new collections module for various specialized collection datatypes. Currently it contains just one type, deque, a double-ended queue that supports efficiently adding and removing elements from either end:

• Several modules, such as the Queue and threading modules, now take advantage of collections.deque for improved performance. (Contributed by Raymond Hettinger.)

• The ConfigParser classes have been enhanced slightly. The read() method now returns a list of the files that were successfully parsed, and the set() method raises TypeError if passed a value argument that isn’t a string. (Contributed by John Belmonte and David Goodger.)

• The curses module now supports the ncurses extension use_default_colors(). On platforms where the terminal supports transparency, this makes it possible to use a transparent background. (Contributed by Jörg Lehmann.)

• The difflib module now includes an HtmlDiff class that creates an HTML table showing a side by side comparison of two versions of a text. (Contributed by Dan Gass.)

• The email package was updated to version 3.0, which dropped various deprecated APIs and removes support for Python versions earlier than 2.3. The 3.0 version of the package uses a new incremental parser for MIME messages, available in the email.FeedParser module. The new parser doesn’t require reading the entire message into memory, and doesn’t raise exceptions if a message is malformed; instead it records any problems in the defect attribute of the message. (Developed by Anthony Baxter, Barry Warsaw, Thomas Wouters, and others.)

• The heapq module has been converted to C. The resulting tenfold improvement in speed makes the module suitable for handling high volumes of data. In addition, the module has two new functions nlargest() and nsmallest() that use heaps to find the N largest or smallest values in a dataset without the expense of a full sort. (Contributed by Raymond Hettinger.)

• The httplib module now contains constants for HTTP status codes defined in various HTTP-related RFC documents. Constants have names such as OK, CREATED, CONTINUE, and MOVED_PERMANENTLY; use pydoc to get a full list. (Contributed by Andrew Eland.)

• The imaplib module now supports IMAP’s THREAD command (contributed by Yves Dionne) and new deleteacl() and myrights() methods (contributed by Arnaud Mazin).

• The itertools module gained a groupby(iterable[, *func]) function. iterable is something that can be iterated over to return a stream of elements, and the optional func parameter is a function that takes an element and returns a key value; if omitted, the key is simply the element itself. groupby() then groups the elements into subsequences which have matching values of the key, and returns a series of 2-tuples containing the key value and an iterator over the subsequence.

  Here’s an example to make this clearer. The key function simply returns whether a number is even or odd, so the result of groupby() is to return consecutive runs of odd or even numbers.

  groupby() is typically used with sorted input. The logic for groupby() is similar to the Unix uniq filter which makes it handy for eliminating, counting, or identifying duplicate elements:

• itertools also gained a function named tee(iterator, N) that returns N independent iterators that replicate iterator. If N is omitted, the default is 2.

  Note that tee() has to keep copies of the values returned by the iterator; in the worst case, it may need to keep all of them. This should therefore be used carefully if the leading iterator can run far ahead of the trailing iterator in a long stream of inputs. If the separation is large, then you might as well use list() instead. When the iterators track closely with one another, tee() is ideal. Possible applications include bookmarking, windowing, or lookahead iterators. (Contributed by Raymond Hettinger.)

• A number of functions were added to the locale module, such as bind_textdomain_codeset() to specify a particular encoding and a family of l*gettext() functions that return messages in the chosen encoding. (Contributed by Gustavo Niemeyer.)

• Some keyword arguments were added to the logging package’s basicConfig() function to simplify log configuration. The default behavior is to log messages to standard error, but various keyword arguments can be specified to log to a particular file, change the logging format, or set the logging level. For example:

• Other additions to the logging package include a log(level, msg) convenience method, as well as a TimedRotatingFileHandler class that rotates its log files at a timed interval. The module already had RotatingFileHandler, which rotated logs once the file exceeded a certain size. Both classes derive from a new BaseRotatingHandler class that can be used to implement other rotating handlers.

• The marshal module now shares interned strings on unpacking a data structure. This may shrink the size of certain pickle strings, but the primary effect is to make .pyc files significantly smaller. (Contributed by Martin von Löwis.)

• The nntplib module’s NNTP class gained description() and descriptions() methods to retrieve newsgroup descriptions for a single group or for a range of groups. (Contributed by Jürgen A. Erhard.)

• Two new functions were added to the operator module, attrgetter(attr) and itemgetter(index). Both functions return callables that take a single argument and return the corresponding attribute or item; these callables make excellent data extractors when used with map() or sorted(). For example:

• The optparse module was updated in various ways. The module now passes its messages through gettext.gettext(), making it possible to internationalize Optik’s help and error messages. Help messages for options can now include the string '%default', which will be replaced by the option’s default value. (Contributed by Greg Ward.)

• The long-term plan is to deprecate the rfc822 module in some future Python release in favor of the email package. To this end, the email.Utils.formatdate function has been changed to make it usable as a replacement for rfc822.formatdate(). You may want to write new e-mail processing code with this in mind. (Change implemented by Anthony Baxter.)

• A new urandom(n) function was added to the os module, returning a string containing n bytes of random data. This function provides access to platform-specific sources of randomness such as /dev/urandom on Linux or the Windows CryptoAPI. (Contributed by Trevor Perrin.)

• Another new function: os.path.lexists(path) returns true if the file specified by path exists, whether or not it’s a symbolic link. This differs from the existing os.path.exists(path) function, which returns false if path is a symlink that points to a destination that doesn’t exist. (Contributed by Beni Cherniavsky.)

• A new getsid() function was added to the posix module that underlies the os module. (Contributed by J. Raynor.)

• The poplib module now supports POP over SSL. (Contributed by Hector Urtubia.)

• The profile module can now profile C extension functions. (Contributed by Nick Bastin.)

• The random module has a new method called getrandbits(N) that returns a long integer N bits in length. The existing randrange() method now uses getrandbits() where appropriate, making generation of arbitrarily large random numbers more efficient. (Contributed by Raymond Hettinger.)

• The regular expression language accepted by the re module was extended with simple conditional expressions, written as (?(group)A|B). group is either a numeric group ID or a group name defined with (?P...) earlier in the expression. If the specified group matched, the regular expression pattern A will be tested against the string; if the group didn’t match, the pattern B will be used instead. (Contributed by Gustavo Niemeyer.)

• The re module is also no longer recursive, thanks to a massive amount of work by Gustavo Niemeyer. In a recursive regular expression engine, certain patterns result in a large amount of C stack space being consumed, and it was possible to overflow the stack. For example, if you matched a 30000-byte string of a characters against the expression (a|b)+, one stack frame was consumed per character. Python 2.3 tried to check for stack overflow and raise a RuntimeError exception, but certain patterns could sidestep the checking and if you were unlucky Python could segfault. Python 2.4’s regular expression engine can match this pattern without problems.

• The signal module now performs tighter error-checking on the parameters to the signal.signal() function. For example, you can’t set a handler on the SIGKILL signal; previous versions of Python would quietly accept this, but 2.4 will raise a RuntimeError exception.

• Two new functions were added to the socket module. socketpair() returns a pair of connected sockets and getservbyport(port) looks up the service name for a given port number. (Contributed by Dave Cole and Barry Warsaw.)

• The sys.exitfunc() function has been deprecated. Code should be using the existing atexit module, which correctly handles calling multiple exit functions. Eventually sys.exitfunc() will become a purely internal interface, accessed only by atexit.

• The tarfile module now generates GNU-format tar files by default. (Contributed by Lars Gustäbel.)

• The threading module now has an elegantly simple way to support thread-local data. The module contains a local class whose attribute values are local to different threads.

  Other threads can assign and retrieve their own values for the number and url attributes. You can subclass local to initialize attributes or to add methods. (Contributed by Jim Fulton.)

• The timeit module now automatically disables periodic garbage collection during the timing loop. This change makes consecutive timings more comparable. (Contributed by Raymond Hettinger.)

• The weakref module now supports a wider variety of objects including Python functions, class instances, sets, frozensets, deques, arrays, files, sockets, and regular expression pattern objects. (Contributed by Raymond Hettinger.)

• The xmlrpclib module now supports a multi-call extension for transmitting multiple XML-RPC calls in a single HTTP operation. (Contributed by Brian Quinlan.)

• The mpz, rotor, and xreadlines modules have been removed.

cookielib

The cookielib library supports client-side handling for HTTP cookies, mirroring the Cookie module’s server-side cookie support. Cookies are stored in cookie jars; the library transparently stores cookies offered by the web server in the cookie jar, and fetches the cookie from the jar when connecting to the server. As in web browsers, policy objects control whether cookies are accepted or not.

In order to store cookies across sessions, two implementations of cookie jars are provided: one that stores cookies in the Netscape format so applications can use the Mozilla or Lynx cookie files, and one that stores cookies in the same format as the Perl libwww library.

urllib2 has been changed to interact with cookielib : HTTPCookieProcessor manages a cookie jar that is used when accessing URLs.

This module was contributed by John J. Lee.

doctest

The doctest module underwent considerable refactoring thanks to Edward Loper and Tim Peters. Testing can still be as simple as running doctest.testmod(), but the refactorings allow customizing the module’s operation in various ways

The new DocTestFinder class extracts the tests from a given object’s docstrings:

The new DocTestRunner class then runs individual tests and can produce a summary of the results:

The above example produces the following output:

DocTestRunner uses an instance of the OutputChecker class to compare the expected output with the actual output. This class takes a number of different flags that customize its behaviour; ambitious users can also write a completely new subclass of OutputChecker.

The default output checker provides a number of handy features. For example, with the doctest.ELLIPSIS option flag, an ellipsis (...) in the expected output matches any substring, making it easier to accommodate outputs that vary in minor ways:

Another special string, , matches a blank line:

Another new capability is producing a diff-style display of the output by specifying the doctest.REPORT_UDIFF (unified diffs), doctest.REPORT_CDIFF (context diffs), or doctest.REPORT_NDIFF (delta-style) option flags. For example:

Build and C API Changes

Some of the changes to Python’s build process and to the C API are:

• Three new convenience macros were added for common return values from extension functions: Py_RETURN_NONE, Py_RETURN_TRUE, and Py_RETURN_FALSE. (Contributed by Brett Cannon.)
• Another new macro, Py_CLEAR, decreases the reference count of obj and sets obj to the null pointer. (Contributed by Jim Fulton.)
• A new function, PyTuple_Pack(N, obj1, obj2, ..., objN), constructs tuples from a variable length argument list of Python objects. (Contributed by Raymond Hettinger.)
• A new function, PyDict_Contains(d, k), implements fast dictionary lookups without masking exceptions raised during the look-up process. (Contributed by Raymond Hettinger.)
• The Py_IS_NAN(X) macro returns 1 if its float or double argument X is a NaN. (Contributed by Tim Peters.)
• C code can avoid unnecessary locking by using the new PyEval_ThreadsInitialized() function to tell if any thread operations have been performed. If this function returns false, no lock operations are needed. (Contributed by Nick Coghlan.)
• A new function, PyArg_VaParseTupleAndKeywords(), is the same as PyArg_ParseTupleAndKeywords() but takes a va_list instead of a number of arguments. (Contributed by Greg Chapman.)
• A new method flag, METH_COEXIST, allows a function defined in slots to co-exist with a PyCFunction having the same name. This can halve the access time for a method such as set.contains(). (Contributed by Raymond Hettinger.)
• Python can now be built with additional profiling for the interpreter itself, intended as an aid to people developing the Python core. Providing --enable-profiling to the configure script will let you profile the interpreter with gprof, and providing the --with-tsc switch enables profiling using the Pentium’s Time-Stamp-Counter register. Note that the --with-tsc switch is slightly misnamed, because the profiling feature also works on the PowerPC platform, though that processor architecture doesn’t call that register “the TSC register”. (Contributed by Jeremy Hylton.)
• The tracebackobject type has been renamed to PyTracebackObject.

Port-Specific Changes

The Windows port now builds under MSVC++ 7.1 as well as version 6. (Contributed by Martin von Löwis.)

Porting to Python 2.4

This section lists previously described changes that may require changes to your code:

• Left shifts and hexadecimal/octal constants that are too large no longer trigger a FutureWarning and return a value limited to 32 or 64 bits; instead they return a long integer.
• Integer operations will no longer trigger an OverflowWarning. The OverflowWarning warning will disappear in Python 2.5.
• The zip() built-in function and itertools.izip() now return an empty list instead of raising a TypeError exception if called with no arguments.
• You can no longer compare the date and datetime instances provided by the datetime module. Two instances of different classes will now always be unequal, and relative comparisons (<, >) will raise a TypeError.
• dircache.listdir() now passes exceptions to the caller instead of returning empty lists.
• LexicalHandler.startDTD() used to receive the public and system IDs in the wrong order. This has been corrected; applications relying on the wrong order need to be fixed.
• fcntl.ioctl() now warns if the mutate argument is omitted and relevant.
• The tarfile module now generates GNU-format tar files by default.
• Encountering a failure while importing a module no longer leaves a partially initialized module object in sys.modules.
• None is now a constant; code that binds a new value to the name None is now a syntax error.
• The signals.signal() function now raises a RuntimeError exception for certain illegal values; previously these errors would pass silently. For example, you can no longer set a handler on the SIGKILL signal.

Acknowledgements

The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Koray Can, Hye-Shik Chang, Michael Dyck, Raymond Hettinger, Brian Hurt, Hamish Lawson, Fredrik Lundh, Sean Reifschneider, Sadruddin Rejeb.

Original source

Python 2.5

This article explains the new features in Python 2.5. The final release of Python 2.5 is scheduled for August 2006; PEP 356 describes the planned release schedule. Python 2.5 was released on September 19, 2006.

The changes in Python 2.5 are an interesting mix of language and library improvements. The library enhancements will be more important to Python’s user community, I think, because several widely useful packages were added. New modules include ElementTree for XML processing (xml.etree), the SQLite database module (sqlite), and the ctypes module for calling C functions.

The language changes are of middling significance. Some pleasant new features were added, but most of them aren’t features that you’ll use every day. Conditional expressions were finally added to the language using a novel syntax; see section PEP 308: Conditional Expressions. The new ‘with’ statement will make writing cleanup code easier (section PEP 343: The ‘with’ statement). Values can now be passed into generators (section PEP 342: New Generator Features). Imports are now visible as either absolute or relative (section PEP 328: Absolute and Relative Imports). Some corner cases of exception handling are handled better (section PEP 341: Unified try/except/finally). All these improvements are worthwhile, but they’re improvements to one specific language feature or another; none of them are broad modifications to Python’s semantics.

As well as the language and library additions, other improvements and bugfixes were made throughout the source tree. A search through the SVN change logs finds there were 353 patches applied and 458 bugs fixed between Python 2.4 and 2.5. (Both figures are likely to be underestimates.)

This article doesn’t try to be a complete specification of the new features; instead changes are briefly introduced using helpful examples. For full details, you should always refer to the documentation for Python 2.5 at https://docs.python.org . If you want to understand the complete implementation and design rationale, refer to the PEP for a particular new feature.

Comments, suggestions, and error reports for this document are welcome; please e-mail them to the author or open a bug in the Python bug tracker.

Original source

This article explains the new features in Python 2.6, released on October 1, 2008. The release schedule is described in PEP 361.

The major theme of Python 2.6 is preparing the migration path to Python 3.0, a major redesign of the language. Whenever possible, Python 2.6 incorporates new features and syntax from 3.0 while remaining compatible with existing code by not removing older features or syntax. When it’s not possible to do that, Python 2.6 tries to do what it can, adding compatibility functions in a future_builtins module and a -3 switch to warn about usages that will become unsupported in 3.0.

Some significant new packages have been added to the standard library, such as the multiprocessing and json modules, but there aren’t many new features that aren’t related to Python 3.0 in some way.

Python 2.6 also sees a number of improvements and bugfixes throughout the source. A search through the change logs finds there were 259 patches applied and 612 bugs fixed between Python 2.5 and 2.6. Both figures are likely to be underestimates.

This article doesn’t attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.6. If you want to understand the rationale for the design and implementation, refer to the PEP for a particular new feature. Whenever possible, “What’s New in Python” links to the bug/patch item for each change.

Original source

Numeric handling has been improved in many ways, for both floating-point numbers and for the Decimal class. There are some useful additions to the standard library, such as a greatly enhanced unittest module, the argparse module for parsing command-line options, convenient OrderedDict and Counter classes in the collections module, and many other improvements.

Python 2.7 is planned to be the last of the 2.x releases, so we worked on making it a good release for the long term. To help with porting to Python 3, several new features from the Python 3.x series have been included in 2.7.

This article doesn’t attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.7 at https://docs.python.org. If you want to understand the rationale for the design and implementation, refer to the PEP for a particular new feature or the issue on https://bugs.python.org in which a change was discussed. Whenever possible, “What’s New in Python” links to the bug/patch item for each change.

Original source

This article explains the new features in Python 3.0, compared to 2.6. Python 3.0, also known as “Python 3000” or “Py3K”, is the first ever intentionally backwards incompatible Python release. Python 3.0 was released on December 3, 2008. There are more changes than in a typical release, and more that are important for all Python users. Nevertheless, after digesting the changes, you’ll find that Python really hasn’t changed all that much – by and large, we’re mostly fixing well-known annoyances and warts, and removing a lot of old cruft.

This article doesn’t attempt to provide a complete specification of all new features, but instead tries to give a convenient overview. For full details, you should refer to the documentation for Python 3.0, and/or the many PEPs referenced in the text. If you want to understand the complete implementation and design rationale for a particular feature, PEPs usually have more details than the regular documentation; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented.

Due to time constraints this document is not as complete as it should have been. As always for a new release, the Misc/NEWS file in the source distribution contains a wealth of detailed information about every small thing that was changed.

Common Stumbling Blocks

This section lists those few changes that are most likely to trip you up if you’re used to Python 2.5.

Print Is A Function

The print statement has been replaced with a print() function, with keyword arguments to replace most of the special syntax of the old print statement (PEP 3105). Examples:

You can also customize the separator between items, e.g.:

which produces:

There are <4294967296> possibilities!

Note:

• The print() function doesn’t support the “softspace” feature of the old print statement. For example, in Python 2.x, print "A\n", "B" would write "A\nB\n"; but in Python 3.0, print("A\n", "B") writes "A\n B\n".
• Initially, you’ll be finding yourself typing the old print x a lot in interactive mode. Time to retrain your fingers to type print(x) instead!
• When using the 2to3 source-to-source conversion tool, all print statements are automatically converted to print() function calls, so this is mostly a non-issue for larger projects.

Views And Iterators Instead Of Lists

Some well-known APIs no longer return lists:

• dict methods dict.keys(), dict.items() and dict.values() return “views” instead of lists. For example, this no longer works: k = d.keys(); k.sort(). Use k = sorted(d) instead (this works in Python 2.5 too and is just as efficient).
• Also, the dict.iterkeys(), dict.iteritems() and dict.itervalues() methods are no longer supported.
• map() and filter() return iterators. If you really need a list and the input sequences are all of equal length, a quick fix is to wrap map() in list(), e.g. list(map(...)), but a better fix is often to use a list comprehension (especially when the original code uses lambda), or rewriting the code so it doesn’t need a list at all. Particularly tricky is map() invoked for the side effects of the function; the correct transformation is to use a regular for loop (since creating a list would just be wasteful).
• If the input sequences are not of equal length, map() will stop at the termination of the shortest of the sequences. For full compatibility with map() from Python 2.x, also wrap the sequences in itertools.zip_longest(), e.g. map(func, *sequences) becomes list(map(func, itertools.zip_longest(*sequences))).
• range() now behaves like xrange() used to behave, except it works with values of arbitrary size. The latter no longer exists.
• zip() now returns an iterator.

Ordering Comparisons

Python 3.0 has simplified the rules for ordering comparisons:

• The ordering comparison operators (<, <=, >=, >) raise a TypeError exception when the operands don’t have a meaningful natural ordering. Thus, expressions like 1 < '', 0 > None or len <= len are no longer valid, and e.g. None < None raises TypeError instead of returning False. A corollary is that sorting a heterogeneous list no longer makes sense – all the elements must be comparable to each other. Note that this does not apply to the == and != operators: objects of different incomparable types always compare unequal to each other.
• sorted() and list.sort() no longer accept the cmp argument providing a comparison function. Use the key argument instead. N.B. the key and reverse arguments are now “keyword-only”.
• The cmp() function should be treated as gone, and the cmp() special method is no longer supported. Use lt() for sorting, eq() with hash(), and other rich comparisons as needed. (If you really need the cmp() functionality, you could use the expression (a > b) - (a < b) as the equivalent for cmp(a, b).)

Integers

• PEP 237: Essentially, long renamed to int. That is, there is only one built-in integral type, named int; but it behaves mostly like the old long type.
• PEP 238: An expression like 1/2 returns a float. Use 1//2 to get the truncating behavior. (The latter syntax has existed for years, at least since Python 2.2.)
• The sys.maxint constant was removed, since there is no longer a limit to the value of integers. However, sys.maxsize can be used as an integer larger than any practical list or string index. It conforms to the implementation’s “natural” integer size and is typically the same as sys.maxint in previous releases on the same platform (assuming the same build options).
• The repr() of a long integer doesn’t include the trailing L anymore, so code that unconditionally strips that character will chop off the last digit instead. (Use str() instead.)
• Octal literals are no longer of the form 0720; use 0o720 instead.

Text Vs. Data Instead Of Unicode Vs. 8-bit

Everything you thought you knew about binary data and Unicode has changed.

• Python 3.0 uses the concepts of text and (binary) data instead of Unicode strings and 8-bit strings. All text is Unicode; however encoded Unicode is represented as binary data. The type used to hold text is str, the type used to hold data is bytes. The biggest difference with the 2.x situation is that any attempt to mix text and data in Python 3.0 raises TypeError, whereas if you were to mix Unicode and 8-bit strings in Python 2.x, it would work if the 8-bit string happened to contain only 7-bit (ASCII) bytes, but you would get UnicodeDecodeError if it contained non-ASCII values. This value-specific behavior has caused numerous sad faces over the years.
• As a consequence of this change in philosophy, pretty much all code that uses Unicode, encodings or binary data most likely has to change. The change is for the better, as in the 2.x world there were numerous bugs having to do with mixing encoded and unencoded text. To be prepared in Python 2.x, start using unicode for all unencoded text, and str for binary or encoded data only. Then the 2to3 tool will do most of the work for you.
• You can no longer use u"..." literals for Unicode text. However, you must use b"..." literals for binary data.
• As the str and bytes types cannot be mixed, you must always explicitly convert between them. Use str.encode() to go from str to bytes, and bytes.decode() to go from bytes to str. You can also use bytes(s, encoding=...) and str(b, encoding=...) , respectively.
• Like str, the bytes type is immutable. There is a separate mutable type to hold buffered binary data, bytearray. Nearly all APIs that accept bytes also accept bytearray. The mutable API is based on collections.MutableSequence.
• All backslashes in raw string literals are interpreted literally. This means that '\U' and '\u' escapes in raw strings are not treated specially. For example, r'\u20ac' is a string of 6 characters in Python 3.0, whereas in 2.6, ur'\u20ac' was the single “euro” character. (Of course, this change only affects raw string literals; the euro character is '\u20ac' in Python 3.0.)
• The built-in basestring abstract type was removed. Use str instead. The str and bytes types don’t have functionality enough in common to warrant a shared base class. The 2to3 tool (see below) replaces every occurrence of basestring with str.
• Files opened as text files (still the default mode for open()) always use an encoding to map between strings (in memory) and bytes (on disk). Binary files (opened with a b in the mode argument) always use bytes in memory. This means that if a file is opened using an incorrect mode or encoding, I/O will likely fail loudly, instead of silently producing incorrect data. It also means that even Unix users will have to specify the correct mode (text or binary) when opening a file. There is a platform-dependent default encoding, which on Unixy platforms can be set with the LANG environment variable (and sometimes also with some other platform-specific locale-related environment variables). In many cases, but not all, the system default is UTF-8; you should never count on this default. Any application reading or writing more than pure ASCII text should probably have a way to override the encoding. There is no longer any need for using the encoding-aware streams in the codecs module.
• The initial values of sys.stdin, sys.stdout, and sys.stderr are now unicode-only text files (i.e., they are instances of io.TextIOBase). To read and write bytes data with these streams, you need to use their io.TextIOBase.buffer attribute.
• Filenames are passed to and returned from APIs as (Unicode) strings. This can present platform-specific problems because on some platforms filenames are arbitrary byte strings. (On the other hand, on Windows filenames are natively stored as Unicode.) As a work-around, most APIs (e.g. open() and many functions in the os module) that take filenames accept bytes objects as well as strings, and a few APIs have a way to ask for a bytes return value. Thus, os.listdir() returns a list of bytes instances if the argument is a bytes instance, and os.getcwdb() returns the current working directory as a bytes instance. Note that when os.listdir() returns a list of strings, filenames that cannot be decoded properly are omitted rather than raising UnicodeError.
• Some system APIs like os.environ and sys.argv can also present problems when the bytes made available by the system is not interpretable using the default encoding. Setting the LANG variable and rerunning the program is probably the best approach.
• PEP 3138: The repr() of a string no longer escapes non-ASCII characters. It still escapes control characters and code points with non-printable status in the Unicode standard, however.
• PEP 3120: The default source encoding is now UTF-8.
• PEP 3131: Non-ASCII letters are now allowed in identifiers. (However, the standard library remains ASCII-only with the exception of contributor names in comments.)
• The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively.
• See also the Unicode HOWTO, which was updated for Python 3.0.

Overview Of Syntax Changes

This section gives a brief overview of every syntactic change in Python 3.0.

New Syntax

• PEP 3107: Function argument and return value annotations. This provides a standardized way of annotating a function’s parameters and return value. There are no semantics attached to such annotations except that they can be introspected at runtime using the annotations attribute. The intent is to encourage experimentation through metaclasses, decorators or frameworks.
• PEP 3102: Keyword-only arguments. Named parameters occurring after *args in the parameter list must be specified using keyword syntax in the call. You can also use a bare * in the parameter list to indicate that you don’t accept a variable-length argument list, but you do have keyword-only arguments.
• Keyword arguments are allowed after the list of base classes in a class definition. This is used by the new convention for specifying a metaclass (see next section), but can be used for other purposes as well, as long as the metaclass supports it.
• PEP 3104: nonlocal statement. Using nonlocal x you can now assign directly to a variable in an outer (but non-global) scope. nonlocal is a new reserved word.
• PEP 3132: Extended Iterable Unpacking. You can now write things like a, b, *rest = some_sequence. And even *rest, a = stuff. The rest object is always a (possibly empty) list; the right-hand side may be any iterable. Example:

This sets a to 0, b to 4, and rest to [1, 2, 3].

• Dictionary comprehensions: {k: v for k, v in stuff} means the same thing as dict(stuff) but is more flexible. (This is PEP 274 vindicated. :-)
• Set literals, e.g. {1, 2}. Note that {} is an empty dictionary; use set() for an empty set. Set comprehensions are also supported; e.g., {x for x in stuff} means the same thing as set(stuff) but is more flexible.
• New octal literals, e.g. 0o720 (already in 2.6). The old octal literals (0720) are gone.
• New binary literals, e.g. 0b1010 (already in 2.6), and there is a new corresponding built-in function, bin().
• Bytes literals are introduced with a leading b or B, and there is a new corresponding built-in function, bytes().

Changed Syntax

• PEP 3109 and PEP 3134: new raise statement syntax: raise [expr [from expr]]. See below.
• as and with are now reserved words. (Since 2.6, actually.)
• True, False, and None are reserved words. (2.6 partially enforced the restrictions on None already.)
• Change from except exc, var to except exc as var. See PEP 3110.
• PEP 3115: New Metaclass Syntax. Instead of:

class C:
    __metaclass__ = M
    ...

you must now use:

class C(metaclass=M):
    ...

The module-global metaclass variable is no longer supported. (It was a crutch to make it easier to default to new-style classes without deriving every class from object.)

• List comprehensions no longer support the syntactic form [... for var in item1, item2, ...]. Use [... for var in (item1, item2, ...)] instead. Also note that list comprehensions have different semantics: they are closer to syntactic sugar for a generator expression inside a list() constructor, and in particular the loop control variables are no longer leaked into the surrounding scope.
• The ellipsis (...) can be used as an atomic expression anywhere. (Previously it was only allowed in slices.) Also, it must now be spelled as ... . (Previously it could also be spelled as . . . , by a mere accident of the grammar.)

Removed Syntax

• PEP 3113: Tuple parameter unpacking removed. You can no longer write def foo(a, (b, c)): ... . Use def foo(a, b_c): b, c = b_c instead.
• Removed backticks (use repr() instead).
• Removed <> (use != instead).
• Removed keyword: exec() is no longer a keyword; it remains as a function. (Fortunately the function syntax was also accepted in 2.x.) Also note that exec() no longer takes a stream argument; instead of exec(f) you can use exec(f.read()).
• Integer literals no longer support a trailing l or L.
• String literals no longer support a leading u or U.
• The from module import * syntax is only allowed at the module level, no longer inside functions.
• The only acceptable syntax for relative imports is from .[module] import name. All import forms not starting with . are interpreted as absolute imports. (PEP 328)
• Classic classes are gone.

Changes Already Present In Python 2.6

Since many users presumably make the jump straight from Python 2.5 to Python 3.0, this section reminds the reader of new features that were originally designed for Python 3.0 but that were back-ported to Python 2.6. The corresponding sections in What’s New in Python 2.6 should be consulted for longer descriptions.

• PEP 343: The ‘with’ statement. The with statement is now a standard feature and no longer needs to be imported from the future. Also check out Writing Context Managers and The contextlib module.
• PEP 366: Explicit Relative Imports From a Main Module. This enhances the usefulness of the -m option when the referenced module lives in a package.
• PEP 370: Per-user site-packages Directory.
• PEP 371: The multiprocessing Package.
• PEP 3101: Advanced String Formatting. Note: the 2.6 description mentions the format() method for both 8-bit and Unicode strings. In 3.0, only the str type (text strings with Unicode support) supports this method; the bytes type does not. The plan is to eventually make this the only API for string formatting, and to start deprecating the % operator in Python 3.1.
• PEP 3105: print As a Function. This is now a standard feature and no longer needs to be imported from future. More details were given above.
• PEP 3110: Exception-Handling Changes. The except exc as var syntax is now standard and except exc, var is no longer supported. (Of course, the as var part is still optional.)
• PEP 3112: Byte Literals. The b"..." string literal notation (and its variants like b'...', b"""...""", and br"...") now produces a literal of type bytes.
• PEP 3116: New I/O Library. The io module is now the standard way of doing file I/O. The built-in open() function is now an alias for io.open() and has additional keyword arguments encoding, errors, newline and closefd. Also note that an invalid mode argument now raises ValueError, not IOError. The binary file object underlying a text file object can be accessed as f.buffer (but beware that the text object maintains a buffer of itself in order to speed up the encoding and decoding operations).
• PEP 3118: Revised Buffer Protocol. The old builtin buffer() is now really gone; the new builtin memoryview() provides (mostly) similar functionality.
• PEP 3119: Abstract Base Classes. The abc module and the ABCs defined in the collections module plays a somewhat more prominent role in the language now, and built-in collection types like dict and list conform to the collections.MutableMapping and collections.MutableSequence ABCs, respectively.
• PEP 3127: Integer Literal Support and Syntax. As mentioned above, the new octal literal notation is the only one supported, and binary literals have been added.
• PEP 3129: Class Decorators.
• PEP 3141: A Type Hierarchy for Numbers. The numbers module is another new use of ABCs, defining Python’s “numeric tower”. Also note the new fractions module which implements numbers.Rational.

Library Changes

Due to time constraints, this document does not exhaustively cover the very extensive changes to the standard library. PEP 3108 is the reference for the major changes to the library. Here’s a capsule review:

• Many old modules were removed. Some, like gopherlib (no longer used) and md5 (replaced by hashlib), were already deprecated by PEP 4. Others were removed as a result of the removal of support for various platforms such as Irix, BeOS and Mac OS 9 (see PEP 11). Some modules were also selected for removal in Python 3.0 due to lack of use or because a better replacement exists. See PEP 3108 for an exhaustive list.
• The bsddb3 package was removed because its presence in the core standard library has proved over time to be a particular burden for the core developers due to testing instability and Berkeley DB’s release schedule. However, the package is alive and well, externally maintained at https://www.jcea.es/programacion/pybsddb.htm.
• Some modules were renamed because their old name disobeyed PEP 8, or for various other reasons. Here’s the list:
  _old->winreg, ConfigParser->configparser, copy_reg->copyreg, Queue->queue, SocketServer->socketserver, markupbase->_markupbase, repr->reprlib, test.test_support->test.support
• A common pattern in Python 2.x is to have one version of a module implemented in pure Python, with an optional accelerated version implemented as a C extension; for example, pickle and cPickle. This places the burden of importing the accelerated version and falling back on the pure Python version on each user of these modules. In Python 3.0, the accelerated versions are considered implementation details of the pure Python versions. Users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version. The pickle/cPickle pair received this treatment. The profile module is on the list for 3.1. The StringIO module has been turned into a class in the io module.
• Some related modules have been grouped into packages, and usually the submodule names have been simplified. The resulting new packages are: dbm (anydbm, dbhash, dbm, dumbdbm, gdbm, whichdb), html (HTMLParser, htmlentitydefs), http (httplib, BaseHTTPServer, CGIHTTPServer, SimpleHTTPServer, Cookie, cookielib), tkinter (all Tkinter-related modules except turtle). The target audience of turtle doesn’t really care about tkinter. Also note that as of Python 2.6, the functionality of turtle has been greatly enhanced. urllib (urllib, urllib2, urlparse, robotparse), xmlrpc (xmlrpclib, DocXMLRPCServer, SimpleXMLRPCServer).
• Some other changes to standard library modules, not covered by PEP 3108:
• Killed sets. Use the built-in set() class.
• Cleanup of the sys module: removed sys.exitfunc(), sys.exc_clear(), sys.exc_type, sys.exc_value, sys.exc_traceback. (Note that sys.last_type etc. remain.)
• Cleanup of the array.array type: the read() and write() methods are gone; use fromfile() and tofile() instead. Also, the 'c' typecode for array is gone – use either 'b' for bytes or 'u' for Unicode characters.
• Cleanup of the operator module: removed sequenceIncludes() and isCallable().
• Cleanup of the thread module: acquire_lock() and release_lock() are gone; use acquire() and release() instead.
• Cleanup of the random module: removed the jumpahead() API.
• The new module is gone.
• The functions os.tmpnam(), os.tempnam() and os.tmpfile() have been removed in favor of the tempfile module.
• The tokenize module has been changed to work with bytes. The main entry point is now tokenize.tokenize(), instead of generate_tokens.
• string.letters and its friends (string.lowercase and string.uppercase) are gone. Use string.ascii_letters etc. instead. (The reason for the removal is that string.letters and friends had locale-specific behavior, which is a bad idea for such attractively named global “constants”.)
• Renamed module builtin to builtins (removing the underscores, adding an ‘s’). The builtins variable found in most global namespaces is unchanged. To modify a builtin, you should use builtins, not builtins!

PEP 3101: A New Approach To String Formatting

A new system for built-in string formatting operations replaces the % string formatting operator. (However, the % operator is still supported; it will be deprecated in Python 3.1 and removed from the language at some later time.) Read PEP 3101 for the full scoop.

Changes To Exceptions

The APIs for raising and catching exception have been cleaned up and new powerful features added:

• PEP 352: All exceptions must be derived (directly or indirectly) from BaseException. This is the root of the exception hierarchy. This is not new as a recommendation, but the requirement to inherit from BaseException is new. (Python 2.6 still allowed classic classes to be raised, and placed no restriction on what you can catch.) As a consequence, string exceptions are finally truly and utterly dead.
• Almost all exceptions should actually derive from Exception; BaseException should only be used as a base class for exceptions that should only be handled at the top level, such as SystemExit or KeyboardInterrupt. The recommended idiom for handling all exceptions except for this latter category is to use except Exception.
• StandardError was removed.
• Exceptions no longer behave as sequences. Use the args attribute instead.
• PEP 3109: Raising exceptions. You must now use raise Exception(args) instead of raise Exception, args. Additionally, you can no longer explicitly specify a traceback; instead, if you have to do this, you can assign directly to the traceback attribute (see below).
• PEP 3110: Catching exceptions. You must now use except SomeException as variable instead of except SomeException, variable. Moreover, the variable is explicitly deleted when the except block is left.
• PEP 3134: Exception chaining. There are two cases: implicit chaining and explicit chaining. Implicit chaining happens when an exception is raised in an except or finally handler block. This usually happens due to a bug in the handler block; we call this a secondary exception. In this case, the original exception (that was being handled) is saved as the context attribute of the secondary exception. Explicit chaining is invoked with this syntax:

raise SecondaryException() from primary_exception

(where primary_exception is any expression that produces an exception object, probably an exception that was previously caught). In this case, the primary exception is stored on the cause attribute of the secondary exception. The traceback printed when an unhandled exception occurs walks the chain of cause and context attributes and prints a separate traceback for each component of the chain, with the primary exception at the top. (Java users may recognize this behavior.)

• PEP 3134: Exception objects now store their traceback as the traceback attribute. This means that an exception object now contains all the information pertaining to an exception, and there are fewer reasons to use sys.exc_info() (though the latter is not removed).
• A few exception messages are improved when Windows fails to load an extension module. For example, error code 193 is now %1 is not a valid Win32 application. Strings now deal with non-English locales.

Miscellaneous Other Changes

Operators And Special Methods

• != now returns the opposite of ==, unless == returns NotImplemented.
• The concept of “unbound methods” has been removed from the language. When referencing a method as a class attribute, you now get a plain function object.
• getslice(), setslice() and delslice() were killed. The syntax a[i:j] now translates to a.getitem(slice(i, j)) (or setitem() or delitem(), when used as an assignment or deletion target, respectively).
• PEP 3114: the standard next() method has been renamed to next().
• The oct() and hex() special methods are removed – oct() and hex() use index() now to convert the argument to an integer.
• Removed support for members and methods.
• The function attributes named func_X have been renamed to use the X form, freeing up these names in the function attribute namespace for user-defined attributes. To wit, func_closure, func_code, func_defaults, func_dict, func_doc, func_globals, func_name were renamed to closure, code, defaults, dict, doc, globals, name, respectively.
• nonzero() is now bool().

Builtins

• PEP 3135: New super(). You can now invoke super() without arguments and (assuming this is in a regular instance method defined inside a class statement) the right class and instance will automatically be chosen. With arguments, the behavior of super() is unchanged.
• PEP 3111: raw_input() was renamed to input(). That is, the new input() function reads a line from sys.stdin and returns it with the trailing newline stripped. It raises EOFError if the input is terminated prematurely. To get the old behavior of input(), use eval(input()).
• A new built-in function next() was added to call the next() method on an object.
• The round() function rounding strategy and return type have changed. Exact halfway cases are now rounded to the nearest even result instead of away from zero. (For example, round(2.5) now returns 2 rather than 3.) round(x[, n]) now delegates to x.round([n]) instead of always returning a float. It generally returns an integer when called with a single argument and a value of the same type as x when called with two arguments.
• Moved intern() to sys.intern().
• Removed apply(). Instead of apply(f, args) use f(*args).
• Removed callable(). Instead of callable(f) you can use isinstance(f, collections.Callable). The operator.isCallable() function is also gone.
• Removed coerce(). This function no longer serves a purpose now that classic classes are gone.
• Removed execfile(). Instead of execfile(fn) use exec(open(fn).read()).
• Removed the file type. Use open(). There are now several different kinds of streams that open can return in the io module.
• Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable.
• Removed reload(). Use imp.reload().
• Removed dict.has_key() – use the in operator instead.

Build and C API Changes

Due to time constraints, here is a very incomplete list of changes to the C API.

• Support for several platforms was dropped, including but not limited to Mac OS 9, BeOS, RISCOS, Irix, and Tru64.
• PEP 3118: New Buffer API.
• PEP 3121: Extension Module Initialization & Finalization.
• PEP 3123: Making PyObject_HEAD conform to standard C.
• No more C API support for restricted execution.
• PyNumber_Coerce(), PyNumber_CoerceEx(), PyMember_Get(), and PyMember_Set() C APIs are removed.
• New C API PyImport_ImportModuleNoBlock(), works like PyImport_ImportModule() but won’t block on the import lock (returning an error instead).
• Renamed the boolean conversion C-level slot and method: nb_nonzero is now nb_bool.
• Removed METH_OLDARGS and WITH_CYCLE_GC from the C API.

Performance

The net result of the 3.0 generalizations is that Python 3.0 runs the pystone benchmark around 10% slower than Python 2.5. Most likely the biggest cause is the removal of special-casing for small integers. There’s room for improvement, but it will happen after 3.0 is released!

Porting To Python 3.0

For porting existing Python 2.5 or 2.6 source code to Python 3.0, the best strategy is the following:

1. (Prerequisite:) Start with excellent test coverage.
2. Port to Python 2.6. This should be no more work than the average port from Python 2.x to Python 2.(x+1). Make sure all your tests pass.
3. (Still using 2.6:) Turn on the -3 command line switch. This enables warnings about features that will be removed (or change) in 3.0. Run your test suite again, and fix code that you get warnings about until there are no warnings left, and all your tests still pass.
4. Run the 2to3 source-to-source translator over your source code tree. Run the result of the translation under Python 3.0. Manually fix up any remaining issues, fixing problems until all tests pass again.

It is not recommended to try to write source code that runs unchanged under both Python 2.6 and 3.0; you’d have to use a very contorted coding style, e.g. avoiding print statements, metaclasses, and much more. If you are maintaining a library that needs to support both Python 2.6 and Python 3.0, the best approach is to modify step 3 above by editing the 2.6 version of the source code and running the 2to3 translator again, rather than editing the 3.0 version of the source code.

For porting C extensions to Python 3.0, please see Porting Extension Modules to Python 3.

Original source

PEP 372: Ordered Dictionaries

Regular Python dictionaries iterate over key/value pairs in arbitrary order. Over the years, a number of authors have written alternative implementations that remember the order that the keys were originally inserted. Based on the experiences from those implementations, a new collections.OrderedDict class has been introduced.

The OrderedDict API is substantially the same as regular dictionaries but will iterate over keys and values in a guaranteed order depending on when a key was first inserted. If a new entry overwrites an existing entry, the original insertion position is left unchanged. Deleting an entry and reinserting it will move it to the end.

The standard library now supports use of ordered dictionaries in several modules. The configparser module uses them by default. This lets configuration files be read, modified, and then written back in their original order. The _asdict() method for collections.namedtuple() now returns an ordered dictionary with the values appearing in the same order as the underlying tuple indices. The json module is being built-out with an object_pairs_hook to allow OrderedDicts to be built by the decoder. Support was also added for third-party tools like PyYAML.

See also

PEP 372 - Ordered Dictionaries

PEP written by Armin Ronacher and Raymond Hettinger. Implementation written by Raymond Hettinger.

Since an ordered dictionary remembers its insertion order, it can be used in conjunction with sorting to make a sorted dictionary:

The new sorted dictionaries maintain their sort order when entries are deleted. But when new keys are added, the keys are appended to the end and the sort is not maintained.

PEP 378: Format Specifier for Thousands Separator

The built-in format() function and the str.format() method use a mini-language that now includes a simple, non-locale aware way to format a number with a thousands separator. That provides a way to humanize a program’s output, improving its professional appearance and readability:

The supported types are int, float, complex and decimal.Decimal.

Discussions are underway about how to specify alternative separators like dots, spaces, apostrophes, or underscores. Locale-aware applications should use the existing n format specifier which already has some support for thousands separators.

See also

PEP 378 - Format Specifier for Thousands Separator

PEP written by Raymond Hettinger and implemented by Eric Smith and Mark Dickinson.

Other Language Changes

Some smaller changes made to the core Python language are:

• Directories and zip archives containing a main.py file can now be executed directly by passing their name to the interpreter. The directory/zipfile is automatically inserted as the first entry in sys.path. (Suggestion and initial patch by Andy Chu; revised patch by Phillip J. Eby and Nick Coghlan; bpo-1739468.)
• The int() type gained a bit_length method that returns the number of bits necessary to represent its argument in binary:
• The fields in format() strings can now be automatically numbered:
• Formerly, the string would have required numbered fields such as: 'Sir {0} of {1}'.
• The string.maketrans() function is deprecated and is replaced by new static methods, bytes.maketrans() and bytearray.maketrans(). This change solves the confusion around which types were supported by the string module. Now, str, bytes, and bytearray each have their own maketrans and translate methods with intermediate translation tables of the appropriate type.
• The syntax of the with statement now allows multiple context managers in a single statement:
• With the new syntax, the contextlib.nested() function is no longer needed and is now deprecated.
• round(x, n) now returns an integer if x is an integer. Previously it returned a float:
• Python now uses David Gay’s algorithm for finding the shortest floating-point representation that doesn’t change its value. This should help mitigate some of the confusion surrounding binary floating-point numbers.
• The significance is easily seen with a number like 1.1 which does not have an exact equivalent in binary floating point. Since there is no exact equivalent, an expression like float('1.1') evaluates to the nearest representable value which is 0x1.199999999999ap+0 in hex or 1.100000000000000088817841970012523233890533447265625 in decimal. That nearest value was and still is used in subsequent floating-point calculations.
• What is new is how the number gets displayed. Formerly, Python used a simple approach. The value of repr(1.1) was computed as format(1.1, '.17g') which evaluated to '1.1000000000000001'. The advantage of using 17 digits was that it relied on IEEE-754 guarantees to assure that eval(repr(1.1)) would round-trip exactly to its original value. The disadvantage is that many people found the output to be confusing (mistaking intrinsic limitations of binary floating-point representation as being a problem with Python itself).
• The new algorithm for repr(1.1) is smarter and returns '1.1'. Effectively, it searches all equivalent string representations (ones that get stored with the same underlying float value) and returns the shortest representation.
• The new algorithm tends to emit cleaner representations when possible, but it does not change the underlying values. So, it is still the case that 1.1 + 2.2 != 3.3 even though the representations may suggest otherwise.
• The new algorithm depends on certain features in the underlying floating-point implementation. If the required features are not found, the old algorithm will continue to be used. Also, the text pickle protocols assure cross-platform portability by using the old algorithm.

New, Improved, and Deprecated Modules

• Added a collections.Counter class to support convenient counting of unique items in a sequence or iterable:
• Added a new module, tkinter.ttk for access to the Tk themed widget set. The basic idea of ttk is to separate, to the extent possible, the code implementing a widget’s behavior from the code implementing its appearance.
• The gzip.GzipFile and bz2.BZ2File classes now support the context management protocol:
• The decimal module now supports methods for creating a decimal object from a binary float. The conversion is exact but can sometimes be surprising:
• The itertools module grew two new functions. The itertools.combinations_with_replacement() function is one of four for generating combinatorics including permutations and Cartesian products. The itertools.compress() function mimics its namesake from APL. Also, the existing itertools.count() function now has an optional step argument and can accept any type of counting sequence including fractions.Fraction and decimal.Decimal:
• collections.namedtuple() now supports a keyword argument rename which lets invalid fieldnames be automatically converted to positional names in the form _0, _1, etc. This is useful when the field names are being created by an external source such as a CSV header, SQL field list, or user input:
• The re.sub(), re.subn(), and re.split() functions now accept a flags parameter.
• The logging module now implements a simple logging.NullHandler class for applications that are not using logging but are calling library code that does. Setting-up a null handler will suppress spurious warnings such as “No handlers could be found for logger foo”:
• The runpy module which supports the -m command line switch now supports the execution of packages by looking for and executing a main submodule when a package name is supplied.
• The pdb module can now access and display source code loaded via zipimport (or any other conformant PEP 302 loader).
• functools.partial objects can now be pickled.
• Add pydoc help topics for symbols so that help('@') works as expected in the interactive environment.
• The unittest module now supports skipping individual tests or classes of tests. And it supports marking a test as an expected failure, a test that is known to be broken, but shouldn’t be counted as a failure on a TestResult:
• Also, tests for exceptions have been builtout to work with context managers using the with statement:
• In addition, several new assertion methods were added including assertSetEqual(), assertDictEqual(), assertDictContainsSubset(), assertListEqual(), assertTupleEqual(), assertSequenceEqual(), assertRaisesRegexp(), assertIsNone(), and assertIsNotNone().
• The io module has three new constants for the seek() method: SEEK_SET, SEEK_CUR, and SEEK_END.
• The sys.version_info tuple is now a named tuple:
• The nntplib and imaplib modules now support IPv6.
• The pickle module has been adapted for better interoperability with Python 2.x when used with protocol 2 or lower. The reorganization of the standard library changed the formal reference for many objects. For example, builtin.set in Python 2 is called builtins.set in Python 3. This change confounded efforts to share data between different versions of Python. But now when protocol 2 or lower is selected, the pickler will automatically use the old Python 2 names for both loading and dumping. This remapping is turned-on by default but can be disabled with the fix_imports option:
• An unfortunate but unavoidable side-effect of this change is that protocol 2 pickles produced by Python 3.1 won’t be readable with Python 3.0. The latest pickle protocol, protocol 3, should be used when migrating data between Python 3.x implementations, as it doesn’t attempt to remain compatible with Python 2.x.
• A new module, importlib was added. It provides a complete, portable, pure Python reference implementation of the import statement and its counterpart, the import() function. It represents a substantial step forward in documenting and defining the actions that take place during imports.

Optimizations

Major performance enhancements have been added:

• The new I/O library (as defined in PEP 3116) was mostly written in Python and quickly proved to be a problematic bottleneck in Python 3.0. In Python 3.1, the I/O library has been entirely rewritten in C and is 2 to 20 times faster depending on the task at hand. The pure Python version is still available for experimentation purposes through the _pyio module.
• Added a heuristic so that tuples and dicts containing only untrackable objects are not tracked by the garbage collector. This can reduce the size of collections and therefore the garbage collection overhead on long-running programs, depending on their particular use of datatypes.
• Enabling a configure option named --with-computed-gotos on compilers that support it (notably: gcc, SunPro, icc), the bytecode evaluation loop is compiled with a new dispatch mechanism which gives speedups of up to 20%, depending on the system, the compiler, and the benchmark.
• The decoding of UTF-8, UTF-16 and LATIN-1 is now two to four times faster.
• The json module now has a C extension to substantially improve its performance. In addition, the API was modified so that json works only with str, not with bytes. That change makes the module closely match the JSON specification which is defined in terms of Unicode.
• Unpickling now interns the attribute names of pickled objects. This saves memory and allows pickles to be smaller.

IDLE

• IDLE’s format menu now provides an option to strip trailing whitespace from a source file.

Build and C API Changes

Changes to Python’s build process and to the C API include:

• Integers are now stored internally either in base 215 or in base 230, the base being determined at build time. Previously, they were always stored in base 215. Using base 230 gives significant performance improvements on 64-bit machines, but benchmark results on 32-bit machines have been mixed. Therefore, the default is to use base 230 on 64-bit machines and base 215 on 32-bit machines; on Unix, there’s a new configure option --enable-big-digits that can be used to override this default.
• Apart from the performance improvements this change should be invisible to end users, with one exception: for testing and debugging purposes there’s a new sys.int_info that provides information about the internal format, giving the number of bits per digit and the size in bytes of the C type used to store each digit:
• The PyLong_AsUnsignedLongLong() function now handles a negative pylong by raising OverflowError instead of TypeError.
• Deprecated PyNumber_Int(). Use PyNumber_Long() instead.
• Added a new PyOS_string_to_double() function to replace the deprecated functions PyOS_ascii_strtod() and PyOS_ascii_atof().
• Added PyCapsule as a replacement for the PyCObject API. The principal difference is that the new type has a well defined interface for passing typing safety information and a less complicated signature for calling a destructor. The old type had a problematic API and is now deprecated.

Porting to Python 3.1

This section lists previously described changes and other bugfixes that may require changes to your code:

• The new floating-point string representations can break existing doctests. For example:
• The automatic name remapping in the pickle module for protocol 2 or lower can make Python 3.1 pickles unreadable in Python 3.0. One solution is to use protocol 3. Another solution is to set the fix_imports option to False. See the discussion above for more details.

Original source

This article explains the new features in Python 3.2 as compared to 3.1. Python 3.2 was released on February 20, 2011. It focuses on a few highlights and gives a few examples. For full details, see the Misc/NEWS file.

See also

• PEP 392 - Python 3.2 Release Schedule

PEP 384: Defining a Stable ABI

In the past, extension modules built for one Python version were often not usable with other Python versions. Particularly on Windows, every feature release of Python required rebuilding all extension modules that one wanted to use. This requirement was the result of the free access to Python interpreter internals that extension modules could use. With Python 3.2, an alternative approach becomes available: extension modules which restrict themselves to a limited API (by defining Py_LIMITED_API) cannot use many of the internals, but are constrained to a set of API functions that are promised to be stable for several releases. As a consequence, extension modules built for 3.2 in that mode will also work with 3.3, 3.4, and so on. Extension modules that make use of details of memory structures can still be built, but will need to be recompiled for every feature release.

PEP 389: Argparse Command Line Parsing Module

A new module for command line parsing, argparse, was introduced to overcome the limitations of optparse which did not provide support for positional arguments (not just options), subcommands, required options and other common patterns of specifying and validating options. This module has already had widespread success in the community as a third-party module. Being more fully featured than its predecessor, the argparse module is now the preferred module for command-line processing. The older module is still being kept available because of the substantial amount of legacy code that depends on it.

Here’s an annotated example parser showing features like limiting results to a set of choices, specifying a metavar in the help screen, validating that one or more positional arguments is present, and making a required option:

Example of calling the parser on a command string:

Example of the parser’s automatically generated help:

An especially nice argparse feature is the ability to define subparsers, each with their own argument patterns and help displays:

See also

• PEP 389 - New Command Line Parsing Module
• PEP written by Steven Bethard.
• Migrating optparse code to argparse for details on the differences from optparse.

PEP 391: Dictionary Based Configuration for Logging

The logging module provided two kinds of configuration, one style with function calls for each option or another style driven by an external file saved in a configparser format. Those options did not provide the flexibility to create configurations from JSON or YAML files, nor did they support incremental configuration, which is needed for specifying logger options from a command line.

To support a more flexible style, the module now offers logging.config.dictConfig() for specifying logging configuration with plain Python dictionaries. The configuration options include formatters, handlers, filters, and loggers. Here’s a working example of a configuration dictionary:

If that dictionary is stored in a file called conf.json, it can be loaded and called with code like this:

See also

• PEP 391 - Dictionary Based Configuration for Logging
• PEP written by Vinay Sajip.

PEP 3148: The concurrent.futures module

Code for creating and managing concurrency is being collected in a new top-level namespace, concurrent. Its first member is a futures package which provides a uniform high-level interface for managing threads and processes.

The design for concurrent.futures was inspired by the java.util.concurrent package. In that model, a running call and its result are represented by a Future object that abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls.

Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management.

Both classes share a common interface with three methods: submit() for scheduling a callable and returning a Future object; map() for scheduling many asynchronous calls at a time, and shutdown() for freeing resources. The class is a context manager and can be used in a with statement to assure that resources are automatically released when currently pending futures are done executing.

A simple of example of ThreadPoolExecutor is a launch of four parallel threads for copying files:

See also

• PEP 3148 - Futures – Execute Computations Asynchronously
• PEP written by Brian Quinlan.
• Code for Threaded Parallel URL reads, an example using threads to fetch multiple web pages in parallel.
• Code for computing prime numbers in parallel, an example demonstrating ProcessPoolExecutor.

PEP 3147: PYC Repository Directories

Python’s scheme for caching bytecode in .pyc files did not work well in environments with multiple Python interpreters. If one interpreter encountered a cached file created by another interpreter, it would recompile the source and overwrite the cached file, thus losing the benefits of caching.

The issue of “pyc fights” has become more pronounced as it has become commonplace for Linux distributions to ship with multiple versions of Python. These conflicts also arise with CPython alternatives such as Unladen Swallow.

To solve this problem, Python’s import machinery has been extended to use distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and Unladen Swallow each competing for a file called “mymodule.pyc”, they will now look for “mymodule.cpython-32.pyc”, “mymodule.cpython-33.pyc”, and “mymodule.unladen10.pyc”. And to prevent all of these new files from cluttering source directories, the pyc files are now collected in a “pycache” directory stored under the package directory.

Aside from the filenames and target directories, the new scheme has a few aspects that are visible to the programmer:

• Imported modules now have a cached attribute which stores the name of the actual file that was imported:
• The tag that is unique to each interpreter is accessible from the imp module:
• Scripts that try to deduce source filename from the imported file now need to be smarter. It is no longer sufficient to simply strip the “c” from a “.pyc” filename. Instead, use the new functions in the imp module:
• The py_compile and compileall modules have been updated to reflect the new naming convention and target directory. The command-line invocation of compileall has new options:
• The importlib.abc module has been updated with new abstract base classes for loading bytecode files. The obsolete ABCs, PyLoader and PyPycLoader, have been deprecated (instructions on how to stay Python 3.1 compatible are included with the documentation).

See also

• PEP 3147 - PYC Repository Directories
• PEP written by Barry Warsaw.

PEP 3149: ABI Version Tagged .so Files

The PYC repository directory allows multiple bytecode cache files to be co-located. This PEP implements a similar mechanism for shared object files by giving them a common directory and distinct names for each version.

The common directory is “pyshared” and the file names are made distinct by identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the major and minor version numbers, and optional build flags (such as “d” for debug, “m” for pymalloc, “u” for wide-unicode). For an arbitrary package “foo”, you may see these files when the distribution package is installed:

In Python itself, the tags are accessible from functions in the sysconfig module:

See also

• PEP 3149 - ABI Version Tagged .so Files
• PEP written by Barry Warsaw.

PEP 3333: Python Web Server Gateway Interface v1.0.1

This informational PEP clarifies how bytes/text issues are to be handled by the WSGI protocol. The challenge is that string handling in Python 3 is most conveniently handled with the str type even though the HTTP protocol is itself bytes oriented.

The PEP differentiates so-called native strings that are used for request/response headers and metadata versus byte strings which are used for the bodies of requests and responses.

The native strings are always of type str but are restricted to code points between U+0000 through U+00FF which are translatable to bytes using Latin-1 encoding. These strings are used for the keys and values in the environment dictionary and for response headers and statuses in the start_response() function. They must follow RFC 2616 with respect to encoding. That is, they must either be ISO-8859-1 characters or use RFC 2047 MIME encoding.

For developers porting WSGI applications from Python 2, here are the salient points:

• If the app already used strings for headers in Python 2, no change is needed.
• If instead, the app encoded output headers or decoded input headers, then the headers will need to be re-encoded to Latin-1. For example, an output header encoded in utf-8 was using h.encode('utf-8') now needs to convert from bytes to native strings using h.encode('utf-8').decode('latin-1').
• Values yielded by an application or sent using the write() method must be byte strings. The start_response() function and environ must use native strings. The two cannot be mixed.

For server implementers writing CGI-to-WSGI pathways or other CGI-style protocols, the users must to be able access the environment using native strings even though the underlying platform may have a different convention. To bridge this gap, the wsgiref module has a new function, wsgiref.handlers.read_environ() for transcoding CGI variables from os.environ into native strings and returning a new dictionary.

See also

• PEP 3333 - Python Web Server Gateway Interface v1.0.1
• PEP written by Phillip Eby.

Other Language Changes

Some smaller changes made to the core Python language are:

• String formatting for format() and str.format() gained new capabilities for the format character #. Previously, for integers in binary, octal, or hexadecimal, it caused the output to be prefixed with ‘0b’, ‘0o’, or ‘0x’ respectively. Now it can also handle floats, complex, and Decimal, causing the output to always have a decimal point even when no digits follow it.
• There is also a new str.format_map() method that extends the capabilities of the existing str.format() method by accepting arbitrary mapping objects. This new method makes it possible to use string formatting with any of Python’s many dictionary-like objects such as defaultdict, Shelf, ConfigParser, or dbm. It is also useful with custom dict subclasses that normalize keys before look-up or that supply a missing method for unknown keys:
• The interpreter can now be started with a quiet option, -q, to prevent the copyright and version information from being displayed in the interactive mode. The option can be introspected using the sys.flags attribute:
• The hasattr() function works by calling getattr() and detecting whether an exception is raised. This technique allows it to detect methods created dynamically by getattr or getattribute which would otherwise be absent from the class dictionary. Formerly, hasattr would catch any exception, possibly masking genuine errors. Now, hasattr has been tightened to only catch AttributeError and let other exceptions pass through:
• The str() of a float or complex number is now the same as its repr(). Previously, the str() form was shorter but that just caused confusion and is no longer needed now that the shortest possible repr() is displayed by default:
• memoryview objects now have a release() method and they also now support the context management protocol. This allows timely release of any resources that were acquired when requesting a buffer from the original object.
• Previously it was illegal to delete a name from the local namespace if it occurs as a free variable in a nested block:
• Struct sequence types are now subclasses of tuple. This means that C structures like those returned by os.stat(), time.gmtime(), and sys.version_info now work like a named tuple and now work with functions and methods that expect a tuple as an argument. This is a big step forward in making the C structures as flexible as their pure Python counterparts:
• Warnings are now easier to control using the PYTHONWARNINGS environment variable as an alternative to using -W at the command line:
• A new warning category, ResourceWarning, has been added. It is emitted when potential issues with resource consumption or cleanup are detected. It is silenced by default in normal release builds but can be enabled through the means provided by the warnings module, or on the command line.
• range objects now support index and count methods. This is part of an effort to make more objects fully implement the collections.Sequence abstract base class. As a result, the language will have a more uniform API. In addition, range objects now support slicing and negative indices, even with values larger than sys.maxsize. This makes range more interoperable with lists:
• The callable() builtin function from Py2.x was resurrected. It provides a concise, readable alternative to using an abstract base class in an expression like isinstance(x, collections.Callable):
• Python’s import mechanism can now load modules installed in directories with non-ASCII characters in the path name. This solved an aggravating problem with home directories for users with non-ASCII characters in their usernames.

New, Improved, and Deprecated Modules

Python’s standard library has undergone significant maintenance efforts and quality improvements.

The biggest news for Python 3.2 is that the email package, mailbox module, and nntplib modules now work correctly with the bytes/text model in Python 3. For the first time, there is correct handling of messages with mixed encodings.

Throughout the standard library, there has been more careful attention to encodings and text versus bytes issues. In particular, interactions with the operating system are now better able to exchange non-ASCII data using the Windows MBCS encoding, locale-aware encodings, or UTF-8.

Another significant win is the addition of substantially better support for SSL connections and security certificates.

In addition, more classes now implement a context manager to support convenient and reliable resource clean-up using a with statement.

aemail

The usability of the email package in Python 3 has been mostly fixed by the extensive efforts of R. David Murray. The problem was that emails are typically read and stored in the form of bytes rather than str text, and they may contain multiple encodings within a single email. So, the email package had to be extended to parse and generate email messages in bytes format.

• New functions message_from_bytes() and message_from_binary_file(), and new classes BytesFeedParser and BytesParser allow binary message data to be parsed into model objects.
• Given bytes input to the model, get_payload() will by default decode a message body that has a Content-Transfer-Encoding of 8bit using the charset specified in the MIME headers and return the resulting string.
• Given bytes input to the model, Generator will convert message bodies that have a Content-Transfer-Encoding of 8bit to instead have a 7bit Content-Transfer-Encoding.
• A new class BytesGenerator produces bytes as output, preserving any unchanged non-ASCII data that was present in the input used to build the model, including message bodies with a Content-Transfer-Encoding of 8bit.
• The smtplib SMTP class now accepts a byte string for the msg argument to the sendmail() method, and a new method, send_message() accepts a Message object and can optionally obtain the from_addr and to_addrs addresses directly from the object.

Original source

This article explains the new features in Python 3.3, compared to 3.2. Python 3.3 was released on September 29, 2012. For full details, see the changelog.

See also

PEP 398 - Python 3.3 Release Schedule

Summary – Release highlights

New syntax features:

• New yield from expression for generator delegation.
• The u'unicode' syntax is accepted again for str objects.

New library modules:

• faulthandler (helps debugging low-level crashes)
• ipaddress (high-level objects representing IP addresses and masks)
• lzma (compress data using the XZ / LZMA algorithm)
• unittest.mock (replace parts of your system under test with mock objects)
• venv (Python virtual environments, as in the popular virtualenv package)

New built-in features:

• Reworked I/O exception hierarchy.

Implementation improvements:

• Rewritten import machinery based on importlib.
• More compact unicode strings.
• More compact attribute dictionaries.

Significantly Improved Library Modules:

• C Accelerator for the decimal module.
• Better unicode handling in the email module (provisional).

Security improvements:

• Hash randomization is switched on by default.

Please read on for a comprehensive list of user-facing changes.

Original source

This article explains the new features in Python 3.5, compared to 3.4. Python 3.5 was released on September 13, 2015. See the changelog for a full list of changes.

See also
PEP 478 - Python 3.5 Release Schedule

Summary – Release highlights
New syntax features:
• PEP 492, coroutines with async and await syntax.
• PEP 465, a new matrix multiplication operator: a @ b.
• PEP 448, additional unpacking generalizations.

New library modules:
• typing: PEP 484 – Type Hints.
• zipapp: PEP 441 Improving Python ZIP Application Support.

New built-in features:
• bytes % args, bytearray % args: PEP 461 – Adding % formatting to bytes and bytearray.
• New bytes.hex(), bytearray.hex() and memoryview.hex() methods. (Contributed by Arnon Yaari in bpo-9951.)
• memoryview now supports tuple indexing (including multi-dimensional). (Contributed by Antoine Pitrou in bpo-23632.)
• Generators have a new gi_yieldfrom attribute, which returns the object being iterated by yield from expressions. (Contributed by Benno Leslie and Yury Selivanov in bpo-24450.)
• A new RecursionError exception is now raised when maximum recursion depth is reached. (Contributed by Georg Brandl in bpo-19235.)

CPython implementation improvements:
• When the LC_TYPE locale is the POSIX locale (C locale), sys.stdin and sys.stdout now use the surrogateescape error handler, instead of the strict error handler. (Contributed by Victor Stinner in bpo-19977.)
• .pyo files are no longer used and have been replaced by a more flexible scheme that includes the optimization level explicitly in .pyc name. (See PEP 488 overview.)
• Builtin and extension modules are now initialized in a multi-phase process, which is similar to how Python modules are loaded. (See PEP 489 overview.)

Significant improvements in the standard library:
• collections.OrderedDict is now implemented in C, which makes it 4 to 100 times faster.
• The ssl module gained support for Memory BIO, which decouples SSL protocol handling from network IO.
• The new os.scandir() function provides a better and significantly faster way of directory traversal.
• functools.lru_cache() has been mostly reimplemented in C, yielding much better performance.
• The new subprocess.run() function provides a streamlined way to run subprocesses.
• The traceback module has been significantly enhanced for improved performance and developer convenience.

Security improvements:
• SSLv3 is now disabled throughout the standard library. It can still be enabled by instantiating a ssl.SSLContext manually. (See bpo-22638 for more details; this change was backported to CPython 3.4 and 2.7.)
• HTTP cookie parsing is now stricter, in order to protect against potential injection attacks. (Contributed by Antoine Pitrou in bpo-22796.)

Windows improvements:
• A new installer for Windows has replaced the old MSI. See Using Python on Windows for more information.
• Windows builds now use Microsoft Visual C++ 14.0, and extension modules should use the same.

Please read on for a comprehensive list of user-facing changes, including many other smaller improvements, CPython optimizations, deprecations, and potential porting issues.

New Features
PEP 492 - Coroutines with async and await syntax
PEP 492 greatly improves support for asynchronous programming in Python by adding awaitable objects, coroutine functions, asynchronous iteration, and asynchronous context managers.
Coroutine functions are declared using the new async def syntax:

async def coro():
... return 'spam'
Inside a coroutine function, the new await expression can be used to suspend coroutine execution until the result is available. Any object can be awaited, as long as it implements the awaitable protocol by defining the await() method.
PEP 492 also adds async for statement for convenient iteration over asynchronous iterables.
An example of a rudimentary HTTP client written using the new syntax:
import asyncio
async def http_get(domain):
reader, writer = await asyncio.open_connection(domain, 80)
writer.write(b'\r\n'.join([b'GET / HTTP/1.1', b'Host: %b' % domain.encode('latin-1'), b'Connection: close', b'', b'']))
async for line in reader:
print('>>>', line)
writer.close()
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(http_get('example.com'))
finally:
loop.close()

Similarly to asynchronous iteration, there is a new syntax for asynchronous context managers. The following script:
import asyncio
async def coro(name, lock):
print('coro {}: waiting for lock'.format(name))
async with lock:
print('coro {}: holding the lock'.format(name))
await asyncio.sleep(1)
print('coro {}: releasing the lock'.format(name))
loop = asyncio.get_event_loop()
lock = asyncio.Lock()
coros = asyncio.gather(coro(1, lock), coro(2, lock))
try:
loop.run_until_complete(coros)
finally:
loop.close()
will output:
coro 2: waiting for lock
coro 2: holding the lock
coro 1: waiting for lock
coro 2: releasing the lock
coro 1: holding the lock
coro 1: releasing the lock
Note that both async for and async with can only be used inside a coroutine function declared with async def.
Coroutine functions are intended to be run inside a compatible event loop, such as the asyncio loop.

Changed in version 3.5.2:
Starting with CPython 3.5.2, aiter can directly return asynchronous iterators. Returning an awaitable object will result in a PendingDeprecationWarning.
See more details in the Asynchronous Iterators documentation section.

See also
PEP 492 – Coroutines with async and await syntax
PEP written and implemented by Yury Selivanov.

PEP 465 - A dedicated infix operator for matrix multiplication
PEP 465 adds the @ infix operator for matrix multiplication. Currently, no builtin Python types implement the new operator, however, it can be implemented by defining matmul(), rmatmul(), and imatmul() for regular, reflected, and in-place matrix multiplication. The semantics of these methods is similar to that of methods defining other infix arithmetic operators.
Matrix multiplication is a notably common operation in many fields of mathematics, science, engineering, and the addition of @ allows writing cleaner code:
S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r)
instead of:
S = dot((dot(H, beta) - r).T, dot(inv(dot(H, V), H.T)), dot(H, beta) - r))
NumPy 1.10 has support for the new operator:

import numpy
x = numpy.ones(3)
x
array([ 1., 1., 1.])

m = numpy.eye(3)
m
array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])

x @ m
array([ 1., 1., 1.])

See also
PEP 465 – A dedicated infix operator for matrix multiplication
PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson.

PEP 448 - Additional Unpacking Generalizations
PEP 448 extends the allowed uses of the * iterable unpacking operator and ** dictionary unpacking operator. It is now possible to use an arbitrary number of unpackings in function calls:

print(*[1], *[2], 3, *[4, 5])
1 2 3 4 5

def fn(a, b, c, d):
... print(a, b, c, d)
...

fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4})
1 2 3 4
Similarly, tuple, list, set, and dictionary displays allow multiple unpackings (see Expression lists and Dictionary displays):

*range(4), 4
(0, 1, 2, 3, 4)

[*range(4), 4]
[0, 1, 2, 3, 4]

{*range(4), 4, *(5, 6, 7)}
{0, 1, 2, 3, 4, 5, 6, 7}

{'x': 1, **{'y': 2}}
{'x': 1, 'y': 2}

See also
PEP 448 – Additional Unpacking Generalizations
PEP written by Joshua Landau; implemented by Neil Girdhar, Thomas Wouters, and Joshua Landau.

PEP 461 - percent formatting support for bytes and bytearray
PEP 461 adds support for the % interpolation operator to bytes and bytearray.
While interpolation is usually thought of as a string operation, there are cases where interpolation on bytes or bytearrays makes sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. This issue is particularly important when dealing with wire format protocols, which are often a mixture of binary and ASCII compatible text.
Examples:

b'Hello %b!' % b'World'
b'Hello World!'

b'x=%i y=%f' % (1, 2.5)
b'x=1 y=2.500000'
Unicode is not allowed for %b, but it is accepted by %a (equivalent of repr(obj).encode('ascii', 'backslashreplace')):

b'Hello %b!' % 'World'
Traceback (most recent call last):
File "", line 1, in
TypeError: %b requires bytes, or an object that implements bytes, not 'str'

b'price: %a' % '10€'
b"price: '10\u20ac'"
Note that %s and %r conversion types, although supported, should only be used in codebases that need compatibility with Python 2.

See also
PEP 461 – Adding % formatting to bytes and bytearray
PEP written by Ethan Furman; implemented by Neil Schemenauer and Ethan Furman.

PEP 484 - Type Hints
Function annotation syntax has been a Python feature since version 3.0 (PEP 3107), however the semantics of annotations has been left undefined.
Experience has shown that the majority of function annotation uses were to provide type hints to function parameters and return values. It became evident that it would be beneficial for Python users, if the standard library included the base definitions and tools for type annotations.
PEP 484 introduces a provisional module to provide these standard definitions and tools, along with some conventions for situations where annotations are not available.
For example, here is a simple function whose argument and return type are declared in the annotations:
def greeting(name: str) -> str:
return 'Hello ' + name
While these annotations are available at runtime through the usual annotations attribute, no automatic type checking happens at runtime. Instead, it is assumed that a separate off-line type checker (e.g. mypy) will be used for on-demand source code analysis.
The type system supports unions, generic types, and a special type named Any which is consistent with (i.e. assignable to and from) all types.

See also
typing module documentation
PEP 484 – Type Hints
PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa; implemented by Guido van Rossum.
PEP 483 – The Theory of Type Hints
PEP written by Guido van Rossum

PEP 471 - os.scandir() function – a better and faster directory iterator
PEP 471 adds a new directory iteration function, os.scandir(), to the standard library. Additionally, os.walk() is now implemented using scandir, which makes it 3 to 5 times faster on POSIX systems and 7 to 20 times faster on Windows systems. This is largely achieved by greatly reducing the number of calls to os.stat() required to walk a directory tree.
Additionally, scandir returns an iterator, as opposed to returning a list of file names, which improves memory efficiency when iterating over very large directories.
The following example shows a simple use of os.scandir() to display all the files (excluding directories) in the given path that don’t start with '.'. The entry.is_file() call will generally not make an additional system call:
for entry in os.scandir(path):
if not entry.name.startswith('.') and entry.is_file():
print(entry.name)

See also
PEP 471 – os.scandir() function – a better and faster directory iterator
PEP written and implemented by Ben Hoyt with the help of Victor Stinner.

PEP 475: Retry system calls failing with EINTR
An errno.EINTR error code is returned whenever a system call, that is waiting for I/O, is interrupted by a signal. Previously, Python would raise InterruptedError in such cases. This meant that, when writing a Python application, the developer had two choices:

1. Ignore the InterruptedError.
2. Handle the InterruptedError and attempt to restart the interrupted system call at every call site.
   The first option makes an application fail intermittently. The second option adds a large amount of boilerplate that makes the code nearly unreadable. Compare:
   print("Hello World")
   and:
   while True:
   try:
   print("Hello World")
   break
   except InterruptedError:
   continue

PEP 475 implements automatic retry of system calls on EINTR. This removes the burden of dealing with EINTR or InterruptedError in user code in most situations and makes Python programs, including the standard library, more robust. Note that the system call is only retried if the signal handler does not raise an exception.
Below is a list of functions which are now retried when interrupted by a signal:
• open() and io.open();
• functions of the faulthandler module;
• os functions: fchdir(), fchmod(), fchown(), fdatasync(), fstat(), fstatvfs(), fsync(), ftruncate(), mkfifo(), mknod(), open(), posix_fadvise(), posix_fallocate(), pread(), pwrite(), read(), readv(), sendfile(), wait3(), wait4(), wait(), waitid(), waitpid(), write(), writev();
• special cases: os.close() and os.dup2() now ignore EINTR errors; the syscall is not retried (see the PEP for the rationale);
• select functions: devpoll.poll(), epoll.poll(), kqueue.control(), poll.poll(), select();
• methods of the socket class: accept(), connect() (except for non-blocking sockets), recv(), recvfrom(), recvmsg(), send(), sendall(), sendmsg(), sendto();
• signal.sigtimedwait() and signal.sigwaitinfo();
• time.sleep().

See also
PEP 475 – Retry system calls failing with EINTR
PEP and implementation written by Charles-François Natali and Victor Stinner, with the help of Antoine Pitrou (the French connection).

PEP 479: Change StopIteration handling inside generators
The interaction of generators and StopIteration in Python 3.4 and earlier was sometimes surprising, and could conceal obscure bugs. Previously, StopIteration raised accidentally inside a generator function was interpreted as the end of the iteration by the loop construct driving the generator.
PEP 479 changes the behavior of generators: when a StopIteration exception is raised inside a generator, it is replaced with a RuntimeError before it exits the generator frame. The main goal of this change is to ease debugging in the situation where an unguarded next() call raises StopIteration and causes the iteration controlled by the generator to terminate silently. This is particularly pernicious in combination with the yield from construct.
This is a backwards incompatible change, so to enable the new behavior, a future import is necessary:

from future import generator_stop
def gen(): ...
next(iter([])) ...
yield ...
next(gen())
Traceback (most recent call last):
File "", line 2, in gen
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
RuntimeError: generator raised StopIteration
Without a future import, a PendingDeprecationWarning will be raised whenever a StopIteration exception is raised inside a generator.

See also
PEP 479 – Change StopIteration handling inside generators
PEP written by Chris Angelico and Guido van Rossum. Implemented by Chris Angelico, Yury Selivanov and Nick Coghlan.

PEP 485: A function for testing approximate equality
PEP 485 adds the math.isclose() and cmath.isclose() functions which tell whether two values are approximately equal or “close” to each other. Whether or not two values are considered close is determined according to given absolute and relative tolerances. Relative tolerance is the maximum allowed difference between isclose arguments, relative to the larger absolute value:

import math
a = 5.0
b = 4.99998
math.isclose(a, b, rel_tol=1e-5)
True

math.isclose(a, b, rel_tol=1e-6)
False
It is also possible to compare two values using absolute tolerance, which must be a non-negative value:

import math
a = 5.0
b = 4.99998
math.isclose(a, b, abs_tol=0.00003)
True

math.isclose(a, b, abs_tol=0.00001)
False

See also
PEP 485 – A function for testing approximate equality
PEP written by Christopher Barker; implemented by Chris Barker and Tal Einat.

PEP 486: Make the Python Launcher aware of virtual environments
PEP 486 makes the Windows launcher (see PEP 397) aware of an active virtual environment. When the default interpreter would be used and the VIRTUAL_ENV environment variable is set, the interpreter in the virtual environment will be used.

See also
PEP 486 – Make the Python Launcher aware of virtual environments
PEP written and implemented by Paul Moore.

PEP 488: Elimination of PYO files
PEP 488 does away with the concept of .pyo files. This means that .pyc files represent both unoptimized and optimized bytecode. To prevent the need to constantly regenerate bytecode files, .pyc files now have an optional opt- tag in their name when the bytecode is optimized. This has the side-effect of no more bytecode file name clashes when running under either -O or -OO . Consequently, bytecode files generated from -O , and -OO may now exist simultaneously. importlib.util.cache_from_source() has an updated API to help with this change.

See also
PEP 488 – Elimination of PYO files
PEP written and implemented by Brett Cannon.

PEP 489: Multi-phase extension module initialization
PEP 489 updates extension module initialization to take advantage of the two step module loading mechanism introduced by PEP 451 in Python 3.4.
This change brings the import semantics of extension modules that opt-in to using the new mechanism much closer to those of Python source and bytecode modules, including the ability to use any valid identifier as a module name, rather than being restricted to ASCII.

See also
PEP 489 – Multi-phase extension module initialization
PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan; implemented by Petr Viktorin.

Other Language Changes
Some smaller changes made to the core Python language are:
• Added the "namereplace" error handlers. The "backslashreplace" error handlers now work with decoding and translating. (Contributed by Serhiy Storchaka in bpo-19676 and bpo-22286.)
• The -b option now affects comparisons of bytes with int. (Contributed by Serhiy Storchaka in bpo-23681.)
• New Kazakh kz1048 and Tajik koi8_t codecs. (Contributed by Serhiy Storchaka in bpo-22682 and bpo-22681.)
• Property docstrings are now writable. This is especially useful for collections.namedtuple() docstrings. (Contributed by Berker Peksag in bpo-24064.)
• Circular imports involving relative imports are now supported. (Contributed by Brett Cannon and Antoine Pitrou in bpo-17636.)

New Modules
typing
The new typing provisional module provides standard definitions and tools for function type annotations. See Type Hints for more information.

zipapp
The new zipapp module (specified in PEP 441) provides an API and command line tool for creating executable Python Zip Applications, which were introduced in Python 2.6 in bpo-1739468, but which were not well publicized, either at the time or since.
With the new module, bundling your application is as simple as putting all the files, including a main.py file, into a directory myapp and running:
$ python -m zipapp myapp
$ python myapp.pyz
The module implementation has been contributed by Paul Moore in bpo-23491.

See also
PEP 441 – Improving Python ZIP Application Support

Improved Modules
argparse
The ArgumentParser class now allows disabling abbreviated usage of long options by setting allow_abbrev to False. (Contributed by Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson in bpo-14910.)

asyncio
Since the asyncio module is provisional, all changes introduced in Python 3.5 have also been backported to Python 3.4.x.
Notable changes in the asyncio module since Python 3.4.0:
• New debugging APIs: loop.set_debug() and loop.get_debug() methods. (Contributed by Victor Stinner.)
• The proactor event loop now supports SSL. (Contributed by Antoine Pitrou and Victor Stinner in bpo-22560.)
• A new loop.is_closed() method to check if the event loop is closed. (Contributed by Victor Stinner in bpo-21326.)
• A new loop.create_task() to conveniently create and schedule a new Task for a coroutine. The create_task method is also used by all asyncio functions that wrap coroutines into tasks, such as asyncio.wait(), asyncio.gather(), etc. (Contributed by Victor Stinner.)
• A new transport.get_write_buffer_limits() method to inquire for high- and low-water limits of the flow control. (Contributed by Victor Stinner.)
• The async() function is deprecated in favor of ensure_future(). (Contributed by Yury Selivanov.)
• New loop.set_task_factory() and loop.get_task_factory() methods to customize the task factory that loop.create_task() method uses. (Contributed by Yury Selivanov.)
• New Queue.join() and Queue.task_done() queue methods. (Contributed by Victor Stinner.)
• The JoinableQueue class was removed, in favor of the asyncio.Queue class. (Contributed by Victor Stinner.)
Updates in 3.5.1:
• The ensure_future() function and all functions that use it, such as loop.run_until_complete(), now accept all kinds of awaitable objects. (Contributed by Yury Selivanov.)
• New run_coroutine_threadsafe() function to submit coroutines to event loops from other threads. (Contributed by Vincent Michel.)
• New Transport.is_closing() method to check if the transport is closing or closed. (Contributed by Yury Selivanov.)
• The loop.create_server() method can now accept a list of hosts. (Contributed by Yann Sionneau.)
Updates in 3.5.2:
• New loop.create_future() method to create Future objects. This allows alternative event loop implementations, such as uvloop, to provide a faster asyncio.Future implementation. (Contributed by Yury Selivanov.)
• New loop.get_exception_handler() method to get the current exception handler. (Contributed by Yury Selivanov.)
• New StreamReader.readuntil() method to read data from the stream until a separator bytes sequence appears. (Contributed by Mark Korenberg.)
• The loop.create_connection() and loop.create_server() methods are optimized to avoid calling the system getaddrinfo function if the address is already resolved. (Contributed by A. Jesse Jiryu Davis.)
• The loop.sock_connect(sock, address) no longer requires the address to be resolved prior to the call. (Contributed by A. Jesse Jiryu Davis.)

bz2
The BZ2Decompressor.decompress method now accepts an optional max_length argument to limit the maximum size of decompressed data. (Contributed by Nikolaus Rath in bpo-15955.)

cgi
The FieldStorage class now supports the context manager protocol. (Contributed by Berker Peksag in bpo-20289.)

cmath
A new function isclose() provides a way to test for approximate equality. (Contributed by Chris Barker and Tal Einat in bpo-24270.)

code
The InteractiveInterpreter.showtraceback() method now prints the full chained traceback, just like the interactive interpreter. (Contributed by Claudiu Popa in bpo-17442.)

collections
The OrderedDict class is now implemented in C, which makes it 4 to 100 times faster. (Contributed by Eric Snow in bpo-16991.)
OrderedDict.items(), OrderedDict.keys(), and OrderedDict.values() views now support reversed() iteration. (Contributed by Serhiy Storchaka in bpo-19505.)
The deque class now defines index(), insert(), and copy(), and supports the + and * operators. This allows deques to be recognized as a MutableSequence and improves their substitutability for lists. (Contributed by Raymond Hettinger in bpo-23704.)
Docstrings produced by namedtuple() can now be updated:
Point = namedtuple('Point', ['x', 'y'])
Point.doc += ': Cartesian coordinate'
Point.x.doc = 'abscissa'
Point.y.doc = 'ordinate'
(Contributed by Berker Peksag in bpo-24064.)
The UserString class now implements the getnewargs(), rmod(), casefold(), format_map(), isprintable(), and maketrans() methods to match the corresponding methods of str. (Contributed by Joe Jevnik in bpo-22189.)

collections.abc
The Sequence.index() method now accepts start and stop arguments to match the corresponding methods of tuple, list, etc. (Contributed by Devin Jeanpierre in bpo-23086.)
A new Generator abstract base class. (Contributed by Stefan Behnel in bpo-24018.)
New Awaitable, Coroutine, AsyncIterator, and AsyncIterable abstract base classes. (Contributed by Yury Selivanov in bpo-24184.)
For earlier Python versions, a backport of the new ABCs is available in an external PyPI package.

compileall
A new compileall option, -j N, allows running N workers simultaneously to perform parallel bytecode compilation. The compile_dir() function has a corresponding workers parameter. (Contributed by Claudiu Popa in bpo-16104.)
Another new option, -r, allows controlling the maximum recursion level for subdirectories. (Contributed by Claudiu Popa in bpo-19628.)
The -q command line option can now be specified more than once, in which case all output, including errors, will be suppressed. The corresponding quiet parameter in compile_dir(), compile_file(), and compile_path() can now accept an integer value indicating the level of output suppression. (Contributed by Thomas Kluyver in bpo-21338.)

concurrent.futures
The Executor.map() method now accepts a chunksize argument to allow batching of tasks to improve performance when ProcessPoolExecutor() is used. (Contributed by Dan O’Reilly in bpo-11271.)
The number of workers in the ThreadPoolExecutor constructor is optional now. The default value is 5 times the number of CPUs. (Contributed by Claudiu Popa in bpo-21527.)

configparser
configparser now provides a way to customize the conversion of values by specifying a dictionary of converters in the ConfigParser constructor, or by defining them as methods in ConfigParser subclasses. Converters defined in a parser instance are inherited by its section proxies.
Example:
import configparser
conv = {}
conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()]
cfg = configparser.ConfigParser(converters=conv)
cfg.read_string("""
[s]
list = a b c d e f g
""")
cfg.get('s', 'list')
'a b c d e f g'
cfg.getlist('s', 'list')
['a', 'b', 'c', 'd', 'e', 'f', 'g']
section = cfg['s']
section.getlist('list')
['a', 'b', 'c', 'd', 'e', 'f', 'g']
(Contributed by Łukasz Langa in bpo-18159.)

contextlib
The new redirect_stderr() context manager (similar to redirect_stdout()) makes it easier for utility scripts to handle inflexible APIs that write their output to sys.stderr and don’t provide any options to redirect it:
import contextlib, io, logging
f = io.StringIO()
with contextlib.redirect_stderr(f):
...
logging.warning('warning')
...
f.getvalue()
'WARNING:root:warning\n'
(Contributed by Berker Peksag in bpo-22389.)

csv
The writerow() method now supports arbitrary iterables, not just sequences. (Contributed by Serhiy Storchaka in bpo-23171.)

curses
The new update_lines_cols() function updates the LINES and COLS module variables. This is useful for detecting manual screen resizing. (Contributed by Arnon Yaari in bpo-4254.)

dbm
dumb.open always creates a new database when the flag has the value "n". (Contributed by Claudiu Popa in bpo-18039.)

difflib
The charset of HTML documents generated by HtmlDiff.make_file() can now be customized by using a new charset keyword-only argument. The default charset of HTML document changed from "ISO-8859-1" to "utf-8". (Contributed by Berker Peksag in bpo-2052.)
The diff_bytes() function can now compare lists of byte strings. This fixes a regression from Python 2. (Contributed by Terry J. Reedy and Greg Ward in bpo-17445.)

distutils
Both the build and build_ext commands now accept a -j option to enable parallel building of extension modules. (Contributed by Antoine Pitrou in bpo-5309.)
The distutils module now supports xz compression, and can be enabled by passing xztar as an argument to bdist --format. (Contributed by Serhiy Storchaka in bpo-16314.)

doctest
The DocTestSuite() function returns an empty unittest.TestSuite if module contains no docstrings, instead of raising ValueError. (Contributed by Glenn Jones in bpo-15916.)
email
A new policy option Policy.mangle_from_ controls whether or not lines that start with "From " in email bodies are prefixed with a ">" character by generators. The default is True for compat32 and False for all other policies. (Contributed by Milan Oberkirch in bpo-20098.)
A new Message.get_content_disposition() method provides easy access to a canonical value for the Content-Disposition header. (Contributed by Abhilash Raj in bpo-21083.)
A new policy option EmailPolicy.utf8 can be set to True to encode email headers using the UTF-8 charset instead of using encoded words. This allows Messages to be formatted according to RFC 6532 and used with an SMTP server that supports the RFC 6531 SMTPUTF8 extension. (Contributed by R. David Murray in bpo-24211.)
The mime.text.MIMEText constructor now accepts a charset.Charset instance. (Contributed by Claude Paroz and Berker Peksag in bpo-16324.)

enum
The Enum callable has a new parameter start to specify the initial number of enum values if only names are provided:

Animal = enum.Enum('Animal', 'cat dog', start=10)
Animal.cat
<Animal.cat: 10>

Animal.dog
<Animal.dog: 11>
(Contributed by Ethan Furman in bpo-21706.)

faulthandler
The enable(), register(), dump_traceback(), and dump_traceback_later() functions now accept file descriptors in addition to file-like objects. (Contributed by Wei Wu in bpo-23566.)

functools
Most of the lru_cache() machinery is now implemented in C, making it significantly faster. (Contributed by Matt Joiner, Alexey Kachayev, and Serhiy Storchaka in bpo-14373.)

glob
The iglob() and glob() functions now support recursive search in subdirectories, using the "**" pattern. (Contributed by Serhiy Storchaka in bpo-13968.)

gzip
The mode argument of the GzipFile constructor now accepts "x" to request exclusive creation. (Contributed by Tim Heaney in bpo-19222.)

heapq
Element comparison in merge() can now be customized by passing a key function in a new optional key keyword argument, and a new optional reverse keyword argument can be used to reverse element comparison:

import heapq
a = ['9', '777', '55555']
b = ['88', '6666']
list(heapq.merge(a, b, key=len))
['9', '88', '777', '6666', '55555']

list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True))
['55555', '6666', '777', '88', '9']
(Contributed by Raymond Hettinger in bpo-13742.)

http
A new HTTPStatus enum that defines a set of HTTP status codes, reason phrases and long descriptions written in English. (Contributed by Demian Brecht in bpo-21793.)

http.client
HTTPConnection.getresponse() now raises a RemoteDisconnected exception when a remote server connection is closed unexpectedly. Additionally, if a ConnectionError (of which RemoteDisconnected is a subclass) is raised, the client socket is now closed automatically, and will reconnect on the next request:
import http.client
conn = http.client.HTTPConnection('www.python.org')
for retries in range(3):
try:
conn.request('GET', '/')
resp = conn.getresponse()
except http.client.RemoteDisconnected:
pass
(Contributed by Martin Panter in bpo-3566.)

idlelib and IDLE
Since idlelib implements the IDLE shell and editor and is not intended for import by other programs, it gets improvements with every release. See Lib/idlelib/NEWS.txt for a cumulative list of changes since 3.4.0, as well as changes made in future 3.5.x releases. This file is also available from the IDLE Help ‣ About IDLE dialog.

imaplib
The IMAP4 class now supports the context manager protocol. When used in a with statement, the IMAP4 LOGOUT command will be called automatically at the end of the block. (Contributed by Tarek Ziadé and Serhiy Storchaka in bpo-4972.)
The imaplib module now supports RFC 5161 (ENABLE Extension) and RFC 6855 (UTF-8 Support) via the IMAP4.enable() method. A new IMAP4.utf8_enabled attribute tracks whether or not RFC 6855 support is enabled. (Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in bpo-21800.)
The imaplib module now automatically encodes non-ASCII string usernames and passwords using UTF-8, as recommended by the RFCs. (Contributed by Milan Oberkirch in bpo-21800.)

imghdr
The what() function now recognizes the OpenEXR format (contributed by Martin Vignali and Claudiu Popa in bpo-20295), and the WebP format (contributed by Fabrice Aneche and Claudiu Popa in bpo-20197.)

importlib
The util.LazyLoader class allows for lazy loading of modules in applications where startup time is important. (Contributed by Brett Cannon in bpo-17621.)
The abc.InspectLoader.source_to_code() method is now a static method. This makes it easier to initialize a module object with code compiled from a string by running exec(code, module.dict). (Contributed by Brett Cannon in bpo-21156.)
The new util.module_from_spec() function is now the preferred way to create a new module. As opposed to creating a types.ModuleType instance directly, this new function will set the various import-controlled attributes based on the passed-in spec object. (Contributed by Brett Cannon in bpo-20383.)

inspect
Both the Signature and Parameter classes are now picklable and hashable. (Contributed by Yury Selivanov in bpo-20726 and bpo-20334.)
A new BoundArguments.apply_defaults() method provides a way to set default values for missing arguments:

def foo(a, b='ham', *args): pass
ba = inspect.signature(foo).bind('spam')
ba.apply_defaults()
ba.arguments
OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())])
(Contributed by Yury Selivanov in bpo-24190.)
A new class method Signature.from_callable() makes subclassing of Signature easier. (Contributed by Yury Selivanov and Eric Snow in bpo-17373.)
The signature() function now accepts a follow_wrapped optional keyword argument, which, when set to False, disables automatic following of wrapped links. (Contributed by Yury Selivanov in bpo-20691.)
A set of new functions to inspect coroutine functions and coroutine objects has been added: iscoroutine(), iscoroutinefunction(), isawaitable(), getcoroutinelocals(), and getcoroutinestate(). (Contributed by Yury Selivanov in bpo-24017 and bpo-24400.)
The stack(), trace(), getouterframes(), and getinnerframes() functions now return a list of named tuples. (Contributed by Daniel Shahaf in bpo-16808.)

io
A new BufferedIOBase.readinto1() method, that uses at most one call to the underlying raw stream’s RawIOBase.read() or RawIOBase.readinto() methods. (Contributed by Nikolaus Rath in bpo-20578.)

ipaddress
Both the IPv4Network and IPv6Network classes now accept an (address, netmask) tuple argument, so as to easily construct network objects from existing addresses:

import ipaddress
ipaddress.IPv4Network(('127.0.0.0', 8))
IPv4Network('127.0.0.0/8')

ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0'))
IPv4Network('127.0.0.0/8')
(Contributed by Peter Moody and Antoine Pitrou in bpo-16531.)
A new reverse_pointer attribute for the IPv4Address and IPv6Address classes returns the name of the reverse DNS PTR record:

import ipaddress
addr = ipaddress.IPv4Address('127.0.0.1')
addr.reverse_pointer
'1.0.0.127.in-addr.arpa'

addr6 = ipaddress.IPv6Address('::1')
addr6.reverse_pointer
'1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'
(Contributed by Leon Weber in bpo-20480.)

json
The json.tool command line interface now preserves the order of keys in JSON objects passed in input. The new --sort-keys option can be used to sort the keys alphabetically. (Contributed by Berker Peksag in bpo-21650.)
JSON decoder now raises JSONDecodeError instead of ValueError to provide better context information about the error. (Contributed by Serhiy Storchaka in bpo-19361.)

linecache
A new lazycache() function can be used to capture information about a non-file-based module to permit getting its lines later via getline(). This avoids doing I/O until a line is actually needed, without having to carry the module globals around indefinitely. (Contributed by Robert Collins in bpo-17911.)

locale
A new delocalize() function can be used to convert a string into a normalized number string, taking the LC_NUMERIC settings into account:

import locale
locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8')
'de_DE.UTF-8'

locale.delocalize('1.234,56')
'1234.56'

locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8')
'en_US.UTF-8'

locale.delocalize('1,234.56')
'1234.56'
(Contributed by Cédric Krier in bpo-13918.)

logging
All logging methods (Logger.log(), exception(), critical(), debug(), etc.), now accept exception instances as an exc_info argument, in addition to boolean values and exception tuples:

import logging
try:
... 1 / 0
... except ZeroDivisionError as ex:
... logging.error('exception', exc_info=ex)
ERROR:root:exception
(Contributed by Yury Selivanov in bpo-20537.)
The handlers.HTTPHandler class now accepts an optional ssl.SSLContext instance to configure SSL settings used in an HTTP connection. (Contributed by Alex Gaynor in bpo-22788.)
The handlers.QueueListener class now takes a respect_handler_level keyword argument which, if set to True, will pass messages to handlers taking handler levels into account. (Contributed by Vinay Sajip.)

lzma
The LZMADecompressor.decompress() method now accepts an optional max_length argument to limit the maximum size of decompressed data. (Contributed by Martin Panter in bpo-15955.)

math
Two new constants have been added to the math module:
inf and nan. (Contributed by Mark Dickinson in bpo-23185.)
A new function isclose() provides a way to test for approximate equality. (Contributed by Chris Barker and Tal Einat in bpo-24270.)
A new gcd() function has been added. The fractions.gcd() function is now deprecated. (Contributed by Mark Dickinson and Serhiy Storchaka in bpo-22486.)

multiprocessing
sharedctypes.synchronized() objects now support the context manager protocol. (Contributed by Charles-François Natali in bpo-21565.)

operator
attrgetter(), itemgetter(), and methodcaller() objects now support pickling. (Contributed by Josh Rosenberg and Serhiy Storchaka in bpo-22955.)
New matmul() and imatmul() functions to perform matrix multiplication. (Contributed by Benjamin Peterson in bpo-21176.)

os
The new scandir() function returning an iterator of DirEntry objects has been added. If possible, scandir() extracts file attributes while scanning a directory, removing the need to perform subsequent system calls to determine file type or attributes, which may significantly improve performance. (Contributed by Ben Hoyt with the help of Victor Stinner in bpo-22524.)
On Windows, a new stat_result.st_file_attributes attribute is now available. It corresponds to the dwFileAttributes member of the BY_HANDLE_FILE_INFORMATION structure returned by GetFileInformationByHandle(). (Contributed by Ben Hoyt in bpo-21719.)
The urandom() function now uses the getrandom() syscall on Linux 3.17 or newer, and getentropy() on OpenBSD 5.6 and newer, removing the need to use /dev/urandom and avoiding failures due to potential file descriptor exhaustion. (Contributed by Victor Stinner in bpo-22181.)
New get_blocking() and set_blocking() functions allow getting and setting a file descriptor’s blocking mode (O_NONBLOCK.) (Contributed by Victor Stinner in bpo-22054.)
The truncate() and ftruncate() functions are now supported on Windows. (Contributed by Steve Dower in bpo-23668.)
There is a new os.path.commonpath() function returning the longest common sub-path of each passed pathname. Unlike the os.path.commonprefix() function, it always returns a valid path:

os.path.commonprefix(['/usr/lib', '/usr/local/lib'])
'/usr/l'

os.path.commonpath(['/usr/lib', '/usr/local/lib'])
'/usr'
(Contributed by Rafik Draoui and Serhiy Storchaka in bpo-10395.)

pathlib
The new Path.samefile() method can be used to check whether the path points to the same file as another path, which can be either another Path object, or a string:

import pathlib
p1 = pathlib.Path('/etc/hosts')
p2 = pathlib.Path('/etc/../etc/hosts')
p1.samefile(p2)
True
(Contributed by Vajrasky Kok and Antoine Pitrou in bpo-19775.)
The Path.mkdir() method now accepts a new optional exist_ok argument to match mkdir -p and os.makedirs() functionality. (Contributed by Berker Peksag in bpo-21539.)
There is a new Path.expanduser() method to expand ~ and ~user prefixes. (Contributed by Serhiy Storchaka and Claudiu Popa in bpo-19776.)
A new Path.home() class method can be used to get a Path instance representing the user’s home directory. (Contributed by Victor Salgado and Mayank Tripathi in bpo-19777.)
New Path.write_text(), Path.read_text(), Path.write_bytes(), Path.read_bytes() methods to simplify read/write operations on files.
The following code snippet will create or rewrite existing file ~/spam42:

import pathlib
p = pathlib.Path('~/spam42')
p.expanduser().write_text('ham')
3
(Contributed by Christopher Welborn in bpo-20218.)

pickle
Nested objects, such as unbound methods or nested classes, can now be pickled using pickle protocols older than protocol version 4. Protocol version 4 already supports these cases. (Contributed by Serhiy Storchaka in bpo-23611.)

poplib
A new POP3.utf8() command enables RFC 6856 (Internationalized Email) support, if a POP server supports it. (Contributed by Milan OberKirch in bpo-21804.)

re
References and conditional references to groups with fixed length are now allowed in lookbehind assertions:

import re
pat = re.compile(r'(a|b).(?<=\1)c')
pat.match('aac')
<_sre.SRE_Match object; span=(0, 3), match='aac'>

pat.match('bbc')
<_sre.SRE_Match object; span=(0, 3), match='bbc'>
(Contributed by Serhiy Storchaka in bpo-9179.)
The number of capturing groups in regular expressions is no longer limited to 100. (Contributed by Serhiy Storchaka in bpo-22437.)
The sub() and subn() functions now replace unmatched groups with empty strings instead of raising an exception. (Contributed by Serhiy Storchaka in bpo-1519638.)
The re.error exceptions have new attributes, msg, pattern, pos, lineno, and colno, that provide better context information about the error:

re.compile("""
... (?x)
... .++
... """)
Traceback (most recent call last):
...
sre_constants.error: multiple repeat at position 16 (line 3, column 7)
(Contributed by Serhiy Storchaka in bpo-22578.)

readline
A new append_history_file() function can be used to append the specified number of trailing elements in history to the given file. (Contributed by Bruno Cauet in bpo-22940.)

selectors
The new DevpollSelector supports efficient /dev/poll polling on Solaris. (Contributed by Giampaolo Rodola’ in bpo-18931.)

shutil
The move() function now accepts a copy_function argument, allowing, for example, the copy() function to be used instead of the default copy2() if there is a need to ignore file metadata when moving. (Contributed by Claudiu Popa in bpo-19840.)
The make_archive() function now supports the xztar format. (Contributed by Serhiy Storchaka in bpo-5411.)

signal
On Windows, the set_wakeup_fd() function now also supports socket handles. (Contributed by Victor Stinner in bpo-22018.)
Various SIG* constants in the signal module have been converted into Enums. This allows meaningful names to be printed during debugging, instead of integer “magic numbers”. (Contributed by Giampaolo Rodola’ in bpo-21076.)

smtpd
Both the SMTPServer and SMTPChannel classes now accept a decode_data keyword argument to determine if the DATA portion of the SMTP transaction is decoded using the "utf-8" codec or is instead provided to the SMTPServer.process_message() method as a byte string. The default is True for backward compatibility reasons, but will change to False in Python 3.6. If decode_data is set to False, the process_message method must be prepared to accept keyword arguments. (Contributed by Maciej Szulik in bpo-19662.)
The SMTPServer class now advertises the 8BITMIME extension (RFC 6152) if decode_data has been set True. If the client specifies BODY=8BITMIME on the MAIL command, it is passed to SMTPServer.process_message() via the mail_options keyword. (Contributed by Milan Oberkirch and R. David Murray in bpo-21795.)
The SMTPServer class now also supports the SMTPUTF8 extension (RFC 6531: Internationalized Email). If the client specified SMTPUTF8 BODY=8BITMIME on the MAIL command, they are passed to SMTPServer.process_message() via the mail_options keyword. It is the responsibility of the process_message method to correctly handle the SMTPUTF8 data. (Contributed by Milan Oberkirch in bpo-21725.)
It is now possible to provide, directly or via name resolution, IPv6 addresses in the SMTPServer constructor, and have it successfully connect. (Contributed by Milan Oberkirch in bpo-14758.)

smtplib
A new SMTP.auth() method provides a convenient way to implement custom authentication mechanisms. (Contributed by Milan Oberkirch in bpo-15014.)
The SMTP.set_debuglevel() method now accepts an additional debuglevel (2), which enables timestamps in debug messages. (Contributed by Gavin Chappell and Maciej Szulik in bpo-16914.)
Both the SMTP.sendmail() and SMTP.send_message() methods now support RFC 6531 (SMTPUTF8). (Contributed by Milan Oberkirch and R. David Murray in bpo-22027.)

sndhdr
The what() and whathdr() functions now return a namedtuple(). (Contributed by Claudiu Popa in bpo-18615.)

socket
Functions with timeouts now use a monotonic clock, instead of a system clock. (Contributed by Victor Stinner in bpo-22043.)
A new socket.sendfile() method allows sending a file over a socket by using the high-performance os.sendfile() function on UNIX, resulting in uploads being from 2 to 3 times faster than when using plain socket.send(). (Contributed by Giampaolo Rodola’ in bpo-17552.)
The socket.sendall() method no longer resets the socket timeout every time bytes are received or sent. The socket timeout is now the maximum total duration to send all data. (Contributed by Victor Stinner in bpo-23853.)
The backlog argument of the socket.listen() method is now optional. By default it is set to SOMAXCONN or to 128, whichever is less. (Contributed by Charles-François Natali in bpo-21455.)

ssl
Memory BIO Support
(Contributed by Geert Jansen in bpo-21965.)
The new SSLObject class has been added to provide SSL protocol support for cases when the network I/O capabilities of SSLSocket are not necessary or are suboptimal. SSLObject represents an SSL protocol instance, but does not implement any network I/O methods, and instead provides a memory buffer interface. The new MemoryBIO class can be used to pass data between Python and an SSL protocol instance.
The memory BIO SSL support is primarily intended to be used in frameworks implementing asynchronous I/O for which SSLSocket’s readiness model (“select/poll”) is inefficient.
A new SSLContext.wrap_bio() method can be used to create a new SSLObject instance.

Application-Layer Protocol Negotiation Support
(Contributed by Benjamin Peterson in bpo-20188.)
Where OpenSSL support is present, the ssl module now implements the Application-Layer Protocol Negotiation TLS extension as described in RFC 7301.
The new SSLContext.set_alpn_protocols() can be used to specify which protocols a socket should advertise during the TLS handshake.
The new SSLSocket.selected_alpn_protocol() returns the protocol that was selected during the TLS handshake. The HAS_ALPN flag indicates whether ALPN support is present.

Other Changes
There is a new SSLSocket.version() method to query the actual protocol version in use. (Contributed by Antoine Pitrou in bpo-20421.)
The SSLSocket class now implements a SSLSocket.sendfile() method. (Contributed by Giampaolo Rodola’ in bpo-17552.)
The SSLSocket.send() method now raises either the ssl.SSLWantReadError or ssl.SSLWantWriteError exception on a non-blocking socket if the operation would block. Previously, it would return 0.
The cert_time_to_seconds() function now interprets the input time as UTC and not as local time, per RFC 5280. Additionally, the return value is always an int. (Contributed by Akira Li in bpo-19940.)
New SSLObject.shared_ciphers() and SSLSocket.shared_ciphers() methods return the list of ciphers sent by the client during the handshake. (Contributed by Benjamin Peterson in bpo-23186.)
The SSLSocket.do_handshake(), SSLSocket.read(), SSLSocket.shutdown(), and SSLSocket.write() methods of the SSLSocket class no longer reset the socket timeout every time bytes are received or sent. The socket timeout is now the maximum total duration of the method. (Contributed by Victor Stinner in bpo-23853.)
The match_hostname() function now supports matching of IP addresses. (Contributed by Antoine Pitrou in bpo-23239.)

sqlite3
The Row class now fully supports the sequence protocol, in particular reversed() iteration and slice indexing. (Contributed by Claudiu Popa in bpo-10203; by Lucas Sinclair, Jessica McKellar, and Serhiy Storchaka in bpo-13583.)

subprocess
The new run() function has been added. It runs the specified command and returns a CompletedProcess object, which describes a finished process. The new API is more consistent and is the recommended approach to invoking subprocesses in Python code that does not need to maintain compatibility with earlier Python versions. (Contributed by Thomas Kluyver in bpo-23342.)
Examples:

subprocess.run(["ls", "-l"])

doesn't capture output

CompletedProcess(args=['ls', '-l'], returncode=0)

subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1

subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n')

sys
A new set_coroutine_wrapper() function allows setting a global hook that will be called whenever a coroutine object is created by an async def function. A corresponding get_coroutine

Original source

This article explains the new features in Python 3.6, compared to 3.5. Python 3.6 was released on December 23, 2016. See the changelog for a full list of changes.

See also
PEP 494 - Python 3.6 Release Schedule

Summary – Release highlights
New syntax features:
• PEP 498, formatted string literals.
• PEP 515, underscores in numeric literals.
• PEP 526, syntax for variable annotations.
• PEP 525, asynchronous generators.
• PEP 530: asynchronous comprehensions.

New library modules:
• secrets: PEP 506 – Adding A Secrets Module To The Standard Library.

CPython implementation improvements:
• The dict type has been reimplemented to use a more compact representation based on a proposal by Raymond Hettinger and similar to the PyPy dict implementation. This resulted in dictionaries using 20% to 25% less memory when compared to Python 3.5.
• Customization of class creation has been simplified with the new protocol.
• The class attribute definition order is now preserved.
• The order of elements in **kwargs now corresponds to the order in which keyword arguments were passed to the function.
• DTrace and SystemTap probing support has been added.
• The new PYTHONMALLOC environment variable can now be used to debug the interpreter memory allocation and access errors.

Significant improvements in the standard library:
• The asyncio module has received new features, significant usability and performance improvements, and a fair amount of bug fixes. Starting with Python 3.6 the asyncio module is no longer provisional and its API is considered stable.
• A new file system path protocol has been implemented to support path-like objects. All standard library functions operating on paths have been updated to work with the new protocol.
• The datetime module has gained support for Local Time Disambiguation.
• The typing module received a number of improvements.
• The tracemalloc module has been significantly reworked and is now used to provide better output for ResourceWarning as well as provide better diagnostics for memory allocation errors. See the PYTHONMALLOC section for more information.

Security improvements:
• The new secrets module has been added to simplify the generation of cryptographically strong pseudo-random numbers suitable for managing secrets such as account authentication, tokens, and similar.
• On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security. See the PEP 524 for the rationale.
• The hashlib and ssl modules now support OpenSSL 1.1.0.
• The default settings and feature set of the ssl module have been improved.
• The hashlib module received support for the BLAKE2, SHA-3 and SHAKE hash algorithms and the scrypt() key derivation function.

Windows improvements:
• PEP 528 and PEP 529, Windows filesystem and console encoding changed to UTF-8.
• The py.exe launcher, when used interactively, no longer prefers Python 2 over Python 3 when the user doesn’t specify a version (via command line arguments or a config file). Handling of shebang lines remains unchanged - “python” refers to Python 2 in that case.
• python.exe and pythonw.exe have been marked as long-path aware, which means that the 260 character path limit may no longer apply. See removing the MAX_PATH limitation for details.
• A ._pth file can be added to force isolated mode and fully specify all search paths to avoid registry and environment lookup. See the documentation for more information.
• A python36.zip file now works as a landmark to infer PYTHONHOME. See the documentation for more information.

PEP 498: Formatted string literals
PEP 498 introduces a new kind of string literals: f-strings, or formatted string literals.
Formatted string literals are prefixed with 'f' and are similar to the format strings accepted by str.format(). They contain replacement fields surrounded by curly braces. The replacement fields are expressions, which are evaluated at run time, and then formatted using the format() protocol:

name = "Fred"

f"He said his name is {name}."
'He said his name is Fred.'

width = 10

precision = 4

value = decimal.Decimal("12.34567")

f"result: {value:{width}.{precision}}" # nested fields
'result: 12.35'

See also
PEP 498 – Literal String Interpolation.
PEP written and implemented by Eric V. Smith.
Feature documentation.

PEP 526: Syntax for variable annotations
PEP 484 introduced the standard for type annotations of function parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating the types of variables including class variables and instance variables:
primes:
List[int]

[]
captain:
str

Note: no initial value!

class
Starship:
stats:
Dict[str, int]

{}
Just as for function annotations, the Python interpreter does not attach any particular meaning to variable annotations and only stores them in the annotations attribute of a class or module.
In contrast to variable declarations in statically typed languages, the goal of annotation syntax is to provide an easy way to specify structured type metadata for third party tools and libraries via the abstract syntax tree and the annotations attribute.

See also
PEP 526 – Syntax for variable annotations.
PEP written by Ryan Gonzalez, Philip House, Ivan Levkivskyi, Lisa Roach, and Guido van Rossum. Implemented by Ivan Levkivskyi.
Tools that use or will use the new syntax: mypy, pytype, PyCharm, etc.

PEP 515: Underscores in Numeric Literals
PEP 515 adds the ability to use underscores in numeric literals for improved readability. For example:

1_000_000_000_000_000
1000000000000000

0x_FF_FF_FF_FF
4294967295
Single underscores are allowed between digits and after any base specifier. Leading, trailing, or multiple underscores in a row are not allowed.
The string formatting language also now has support for the '_' option to signal the use of an underscore for a thousands separator for floating-point presentation types and for integer presentation type 'd'. For integer presentation types 'b', 'o', 'x', and 'X', underscores will be inserted every 4 digits:

'{:_}'.format(1000000)
'1_000_000'

'{:_x}'.format(0xFFFFFFFF)
'ffff_ffff'

See also
PEP 515 – Underscores in Numeric Literals
PEP written by Georg Brandl and Serhiy Storchaka.

PEP 525: Asynchronous Generators
PEP 492 introduced support for native coroutines and async/await syntax to Python 3.5. A notable limitation of the Python 3.5 implementation is that it was not possible to use await and yield in the same function body. In Python 3.6 this restriction has been lifted, making it possible to define asynchronous generators:
async def ticker(delay, to):
"""Yield numbers from 0 to to every delay seconds."""
for i in range(to):
yield i
await asyncio.sleep(delay)
The new syntax allows for faster and more concise code.

See also
PEP 525 – Asynchronous Generators
PEP written and implemented by Yury Selivanov.

PEP 530: Asynchronous Comprehensions
PEP 530 adds support for using async for in list, set, dict comprehensions and generator expressions:
result = [i async for i in aiter() if i % 2]
Additionally, await expressions are supported in all kinds of comprehensions:
result = [await fun() for fun in funcs if await condition()]

See also
PEP 530 – Asynchronous Comprehensions
PEP written and implemented by Yury Selivanov.

PEP 487: Simpler customization of class creation
It is now possible to customize subclass creation without using a metaclass. The new init_subclass classmethod will be called on the base class whenever a new subclass is created:
class PluginBase:
subclasses = []
def init_subclass(cls, **kwargs):
super().init_subclass(**kwargs)
cls.subclasses.append(cls)
class Plugin1(PluginBase):
pass
class Plugin2(PluginBase):
pass
In order to allow zero-argument super() calls to work correctly from init_subclass() implementations, custom metaclasses must ensure that the new classcell namespace entry is propagated to type.new (as described in Creating the class object).

See also
PEP 487 – Simpler customization of class creation
PEP written and implemented by Martin Teichmann.
Feature documentation.

PEP 487: Descriptor Protocol Enhancements
PEP 487 extends the descriptor protocol to include the new optional set_name() method. Whenever a new class is defined, the new method will be called on all descriptors included in the definition, providing them with a reference to the class being defined and the name given to the descriptor within the class namespace. In other words, instances of descriptors can now know the attribute name of the descriptor in the owner class:
class IntField:
def get(self, instance, owner):
return instance.dict[self.name]
def set(self, instance, value):
if not isinstance(value, int):
raise ValueError(f'expecting integer in {self.name}')
instance.dict[self.name] = value

this is the new initializer:

def set_name(self, owner, name):
self.name = name
class Model:
int_field = IntField()

See also
PEP 487 – Simpler customization of class creation
PEP written and implemented by Martin Teichmann.
Feature documentation.

PEP 519: Adding a file system path protocol
File system paths have historically been represented as str or bytes objects. This has led to people who write code which operate on file system paths to assume that such objects are only one of those two types (an int representing a file descriptor does not count as that is not a file path). Unfortunately that assumption prevents alternative object representations of file system paths like pathlib from working with pre-existing code, including Python’s standard library.
To fix this situation, a new interface represented by os.PathLike has been defined. By implementing the fspath() method, an object signals that it represents a path. An object can then provide a low-level representation of a file system path as a str or bytes object. This means an object is considered path-like if it implements os.PathLike or is a str or bytes object which represents a file system path. Code can use os.fspath(), os.fsdecode(), or os.fsencode() to explicitly get a str and/or bytes representation of a path-like object.
The built-in open() function has been updated to accept os.PathLike objects, as have all relevant functions in the os and os.path modules, and most other functions and classes in the standard library. The os.DirEntry class and relevant classes in pathlib have also been updated to implement os.PathLike.
The hope is that updating the fundamental functions for operating on file system paths will lead to third-party code to implicitly support all path-like objects without any code changes, or at least very minimal ones (e.g. calling os.fspath() at the beginning of code before operating on a path-like object).
Here are some examples of how the new interface allows for pathlib.Path to be used more easily and transparently with pre-existing code:

import pathlib

with open(pathlib.Path("README")) as f:
...
contents = f.read()
...

import os.path

os.path.splitext(pathlib.Path("some_file.txt"))
('some_file', '.txt')

os.path.join("/a/b", pathlib.Path("c"))
'/a/b/c'

import os

os.fspath(pathlib.Path("some_file.txt"))
'some_file.txt'
(Implemented by Brett Cannon, Ethan Furman, Dusty Phillips, and Jelle Zijlstra.)

See also
PEP 519 – Adding a file system path protocol
PEP written by Brett Cannon and Koos Zevenhoven.

PEP 495: Local Time Disambiguation
In most world locations, there have been and will be times when local clocks are moved back. In those times, intervals are introduced in which local clocks show the same time twice in the same day. In these situations, the information displayed on a local clock (or stored in a Python datetime instance) is insufficient to identify a particular moment in time.
PEP 495 adds the new fold attribute to instances of datetime.datetime and datetime.time classes to differentiate between two moments in time for which local times are the same:

u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc)

for i in range(4):
...
u = u0 + i * HOUR
...
t = u.astimezone(Eastern)
...
print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold)
...
04:00:00 UTC = 00:00:00 EDT 0
05:00:00 UTC = 01:00:00 EDT 0
06:00:00 UTC = 01:00:00 EST 1
07:00:00 UTC = 02:00:00 EST 0
The values of the fold attribute have the value 0 for all instances except those that represent the second (chronologically) moment in time in an ambiguous case.

See also
PEP 495 – Local Time Disambiguation
PEP written by Alexander Belopolsky and Tim Peters, implementation by Alexander Belopolsky.

PEP 529: Change Windows filesystem encoding to UTF-8
Representing filesystem paths is best performed with str (Unicode) rather than bytes. However, there are some situations where using bytes is sufficient and correct.
Prior to Python 3.6, data loss could result when using bytes paths on Windows. With this change, using bytes to represent paths is now supported on Windows, provided those bytes are encoded with the encoding returned by sys.getfilesystemencoding(), which now defaults to 'utf-8'.
Applications that do not use str to represent paths should use os.fsencode() and os.fsdecode() to ensure their bytes are correctly encoded. To revert to the previous behaviour, set PYTHONLEGACYWINDOWSFSENCODING or call sys._enablelegacywindowsfsencoding().
See PEP 529 for more information and discussion of code modifications that may be required.

PEP 528: Change Windows console encoding to UTF-8
The default console on Windows will now accept all Unicode characters and provide correctly read str objects to Python code. sys.stdin, sys.stdout, and sys.stderr now default to utf-8 encoding.
This change only applies when using an interactive console, and not when redirecting files or pipes. To revert to the previous behaviour for interactive console use, set PYTHONLEGACYWINDOWSSTDIO.

See also
PEP 528 – Change Windows console encoding to UTF-8
PEP written and implemented by Steve Dower.

PEP 520: Preserving Class Attribute Definition Order
Attributes in a class definition body have a natural ordering: the same order in which the names appear in the source. This order is now preserved in the new class’s dict attribute.
Also, the effective default class execution namespace (returned from type.prepare() ) is now an insertion-order-preserving mapping.

See also
PEP 520 – Preserving Class Attribute Definition Order
PEP written and implemented by Eric Snow.

PEP 468: Preserving Keyword Argument Order
**kwargs in a function signature is now guaranteed to be an insertion-order-preserving mapping.

See also
PEP 468 – Preserving Keyword Argument Order
PEP written and implemented by Eric Snow.

New dict implementation
New dict implementation
The dict type now uses a “compact” representation based on a proposal by Raymond Hettinger which was first implemented by PyPy. The memory usage of the new dict() is between 20% and 25% smaller compared to Python 3.5.
The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5).
(Contributed by INADA Naoki in bpo-27350. Idea originally suggested by Raymond Hettinger.)

PEP 523: Adding a frame evaluation API to CPython
While Python provides extensive support to customize how code executes, one place it has not done so is in the evaluation of frame objects. If you wanted some way to intercept frame evaluation in Python there really wasn’t any way without directly manipulating function pointers for defined functions.
PEP 523 changes this by providing an API to make frame evaluation pluggable at the C level. This will allow for tools such as debuggers and JITs to intercept frame evaluation before the execution of Python code begins. This enables the use of alternative evaluation implementations for Python code, tracking frame evaluation, etc.
This API is not part of the limited C API and is marked as private to signal that usage of this API is expected to be limited and only applicable to very select, low-level use-cases. Semantics of the API will change with Python as necessary.

See also
PEP 523 – Adding a frame evaluation API to CPython
PEP written by Brett Cannon and Dino Viehland.

PYTHONMALLOC environment variable
The new PYTHONMALLOC environment variable allows setting the Python memory allocators and installing debug hooks.
It is now possible to install debug hooks on Python memory allocators on Python compiled in release mode using PYTHONMALLOC=debug. Effects of debug hooks:
• Newly allocated memory is filled with the byte 0xCB
• Freed memory is filled with the byte 0xDB
• Detect violations of the Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc().
• Detect writes before the start of a buffer (buffer underflows)
• Detect writes after the end of a buffer (buffer overflows)
• Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc() ) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc() ) domains are called.
Checking if the GIL is held is also a new feature of Python 3.6.
See the PyMem_SetupDebugHooks() function for debug hooks on Python memory allocators.
It is now also possible to force the usage of the malloc() allocator of the C library for all Python memory allocations using PYTHONMALLOC=malloc. This is helpful when using external memory debuggers like Valgrind on a Python compiled in release mode.
On error, the debug hooks on Python memory allocators now use the tracemalloc module to get the traceback where a memory block was allocated.
Example of fatal error on buffer overflow using python3.6 -X tracemalloc=5 (store 5 frames in traces):
Debug memory block at address p = 0x7fbcd41666f8: API 'o' 4 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail = 0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb): at tail+0: 0x02 *** OUCH at tail+1: 0xfb at tail+2: 0xfb at tail+3: 0xfb at tail+4: 0xfb at tail+5: 0xfb at tail+6: 0xfb at tail+7: 0xfb The block was made by call #1233329 to debug malloc/realloc. Data at p: 1 a 2 b 30 00 Memory block allocated at (most recent call first): File "test/test_bytes.py", line 323 File "unittest/case.py", line 600 File "unittest/case.py", line 648 File "unittest/suite.py", line 122 File "unittest/suite.py", line 84 Fatal Python error : bad trailing pad byte Current thread 0x00007fbcdbd32700 (most recent call first): File "test/test_bytes.py", line 323 in test_hex File "unittest/case.py", line 600 in run File "unittest/case.py", line 648 in call File "unittest/suite.py", line 122 in run File "unittest/suite.py", line 84 in call ...
(Contributed by Victor Stinner in bpo-26516 and bpo-26564.)

DTrace and SystemTap probing support
Python can now be built --with-dtrace which enables static markers for the following events in the interpreter:
• function call/return
• garbage collection started/finished
• line of code executed.
This can be used to instrument running interpreters in production, without the need to recompile specific debug builds or providing application-specific profiling/debugging code.
More details in Instrumenting CPython with DTrace and SystemTap.
The current implementation is tested on Linux and macOS. Additional markers may be added in the future.
(Contributed by Łukasz Langa in bpo-21590, based on patches by Jesús Cea Avión, David Malcolm, and Nikhil Benesch.)

Other Language Changes
Some smaller changes made to the core Python language are:
• A global or nonlocal statement must now textually appear before the first use of the affected name in the same scope. Previously this was a SyntaxWarning.
• It is now possible to set a special method to None to indicate that the corresponding operation is not available. For example, if a class sets iter() to None, the class is not iterable. (Contributed by Andrew Barnert and Ivan Levkivskyi in bpo-25958.)
• Long sequences of repeated traceback lines are now abbreviated as "[Previous line repeated {count} more times]" (see traceback for an example). (Contributed by Emanuel Barry in bpo-26823.)
• Import now raises the new exception ModuleNotFoundError (subclass of ImportError) when it cannot find a module. Code that currently checks for ImportError (in try-except) will still work. (Contributed by Eric Snow in bpo-15767.)
• Class methods relying on zero-argument super() will now work correctly when called from metaclass methods during class creation. (Contributed by Martin Teichmann in bpo-23722.)

New Modules
secrets
The main purpose of the new secrets module is to provide an obvious way to reliably generate cryptographically strong pseudo-random values suitable for managing secrets, such as account authentication, tokens, and similar.
Warning
Note that the pseudo-random generators in the random module should NOT be used for security purposes. Use secrets on Python 3.6+ and os.urandom() on Python 3.5 and earlier.

See also
PEP 506 – Adding A Secrets Module To The Standard Library
PEP written and implemented by Steven D’Aprano.

Improved Modules
array
Exhausted iterators of array.array will now stay exhausted even if the iterated array is extended. This is consistent with the behavior of other mutable sequences.
Contributed by Serhiy Storchaka in bpo-26492.

ast
The new ast.Constant AST node has been added. It can be used by external AST optimizers for the purposes of constant folding.
Contributed by Victor Stinner in bpo-26146.

asyncio
Starting with Python 3.6 the asyncio module is no longer provisional and its API is considered stable.
Notable changes in the asyncio module since Python 3.5.0 (all backported to 3.5.x due to the provisional status):
• The get_event_loop() function has been changed to always return the currently running loop when called from coroutines and callbacks. (Contributed by Yury Selivanov in bpo-28613.)
• The ensure_future() function and all functions that use it, such as loop.run_until_complete(), now accept all kinds of awaitable objects. (Contributed by Yury Selivanov.)
• New run_coroutine_threadsafe() function to submit coroutines to event loops from other threads. (Contributed by Vincent Michel.)
• New Transport.is_closing() method to check if the transport is closing or closed. (Contributed by Yury Selivanov.)
• The loop.create_server() method can now accept a list of hosts. (Contributed by Yann Sionneau.)
• New loop.create_future() method to create Future objects. This allows alternative event loop implementations, such as uvloop, to provide a faster asyncio.Future implementation. (Contributed by Yury Selivanov in bpo-27041.)
• New loop.get_exception_handler() method to get the current exception handler. (Contributed by Yury Selivanov in bpo-27040.)
• New StreamReader.readuntil() method to read data from the stream until a separator bytes sequence appears. (Contributed by Mark Korenberg.)
• The performance of StreamReader.readexactly() has been improved. (Contributed by Mark Korenberg in bpo-28370.)
• The loop.getaddrinfo() method is optimized to avoid calling the system getaddrinfo function if the address is already resolved. (Contributed by A. Jesse Jiryu Davis.)
• The loop.stop() method has been changed to stop the loop immediately after the current iteration. Any new callbacks scheduled as a result of the last iteration will be discarded. (Contributed by Guido van Rossum in bpo-25593.)
• Future.set_exception will now raise TypeError when passed an instance of the StopIteration exception. (Contributed by Chris Angelico in bpo-26221.)
• New loop.connect_accepted_socket() method to be used by servers that accept connections outside of asyncio, but that use asyncio to handle them. (Contributed by Jim Fulton in bpo-27392.)
• TCP_NODELAY flag is now set for all TCP transports by default. (Contributed by Yury Selivanov in bpo-27456.)
• New loop.shutdown_asyncgens() to properly close pending asynchronous generators before closing the loop. (Contributed by Yury Selivanov in bpo-28003.)
• Future and Task classes now have an optimized C implementation which makes asyncio code up to 30% faster. (Contributed by Yury Selivanov and INADA Naoki in bpo-26081 and bpo-28544.)

binascii
The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline character is appended to the return value. (Contributed by Victor Stinner in bpo-25357.)

cmath
The new cmath.tau (τ) constant has been added. (Contributed by Lisa Roach in bpo-12345, see PEP 628 for details.)
New constants: cmath.inf and cmath.nan to match math.inf and math.nan, and also cmath.infj and cmath.nanj to match the format used by complex repr. (Contributed by Mark Dickinson in bpo-23229.)

collections
The new Collection abstract base class has been added to represent sized iterable container classes. (Contributed by Ivan Levkivskyi, docs by Neil Girdhar in bpo-27598.)
The new Reversible abstract base class represents iterable classes that also provide the reversed() method. (Contributed by Ivan Levkivskyi in bpo-25987.)
The new AsyncGenerator abstract base class represents asynchronous generators. (Contributed by Yury Selivanov in bpo-28720.)
The namedtuple() function now accepts an optional keyword argument module , which, when specified, is used for the module attribute of the returned named tuple class. (Contributed by Raymond Hettinger in bpo-17941.)
The verbose and rename arguments for namedtuple() are now keyword-only. (Contributed by Raymond Hettinger in bpo-25628.)
Recursive collections.deque instances can now be pickled. (Contributed by Serhiy Storchaka in bpo-26482.)

concurrent.futures
The ThreadPoolExecutor class constructor now accepts an optional thread_name_prefix argument to make it possible to customize the names of the threads created by the pool. (Contributed by Gregory P. Smith in bpo-27664.)

contextlib
The contextlib.AbstractContextManager class has been added to provide an abstract base class for context managers. It provides a sensible default implementation for enter() which returns self and leaves exit() an abstract method. A matching class has been added to the typing module as typing.ContextManager . (Contributed by Brett Cannon in bpo-25609.)

datetime
The datetime and time classes have the new fold attribute used to disambiguate local time when necessary. Many functions in the datetime have been updated to support local time disambiguation. See Local Time Disambiguation section for more information. (Contributed by Alexander Belopolsky in bpo-24773.)
The datetime.strftime() and date.strftime() methods now support ISO 8601 date directives %G , %u and %V . (Contributed by Ashley Anderson in bpo-12006.)
The datetime.isoformat() function now accepts an optional timespec argument that specifies the number of additional components of the time value to include. (Contributed by Alessandro Cucci and Alexander Belopolsky in bpo-19475.)
The datetime.combine() now accepts an optional tzinfo argument. (Contributed by Alexander Belopolsky in bpo-27661.)

decimal
New Decimal.as_integer_ratio() method that returns a pair (n, d) of integers that represent the given Decimal instance as a fraction, in lowest terms and with a positive denominator:

Decimal('-3.14').as_integer_ratio()
(-157, 50)
(Contributed by Stefan Krah amd Mark Dickinson in bpo-25928.)

distutils
The default_format attribute has been removed from distutils.command.sdist.sdist and the formats attribute defaults to ['gztar'] . Although not anticipated, any code relying on the presence of default_format may need to be adapted. See bpo-27819 for more details.

email
The new email API, enabled via the policy keyword to various constructors, is no longer provisional. The email documentation has been reorganized and rewritten to focus on the new API, while retaining the old documentation for the legacy API. (Contributed by R. David Murray in bpo-24277.)
The email.mime classes now all accept an optional policy keyword. (Contributed by Berker Peksag in bpo-27331.)
The DecodedGenerator now supports the policy keyword.
There is a new policy attribute, message_factory , that controls what class is used by default when the parser creates new message objects. For the email.policy.compat32 policy this is Message , for the new policies it is EmailMessage . (Contributed by R. David Murray in bpo-20476.)

encodings
On Windows, added the 'oem' encoding to use CP_OEMCP , and the 'ansi' alias for the existing 'mbcs' encoding, which uses the CP_ACP code page. (Contributed by Steve Dower in bpo-27959.)

enum
Two new enumeration base classes have been added to the enum module: Flag and IntFlag . Both are used to define constants that can be combined using the bitwise operators. (Contributed by Ethan Furman in bpo-23591.)
Many standard library modules have been updated to use the IntFlag class for their constants.
The new enum.auto value can be used to assign values to enum members automatically:

from enum import Enum, auto

class Color(Enum):
...
red = auto()
...
blue = auto()
...
green = auto()

list(Color)
[<Color.red: 1>, <Color.blue: 2>, <Color.green: 3>]

faulthandler
On Windows, the faulthandler module now installs a handler for Windows exceptions: see faulthandler.enable() . (Contributed by Victor Stinner in bpo-23848.)

fileinput
hook_encoded() now supports the errors argument. (Contributed by Joseph Hackman in bpo-25788.)

hashlib
hashlib supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in bpo-26470.)
BLAKE2 hash functions were added to the module. blake2b() and blake2s() are always available and support the full feature set of BLAKE2. (Contributed by Christian Heimes in bpo-26798 based on code by Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.)
The SHA-3 hash functions sha3_224() , sha3_256() , sha3_384() , sha3_512() , and SHAKE hash functions shake_128() and shake_256() were added. (Contributed by Christian Heimes in bpo-16113. Keccak Code Package by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and Ronny Van Keer.)
The password-based key derivation function scrypt() is now available with OpenSSL 1.1.0 and newer. (Contributed by Christian Heimes in bpo-27928.)

http.client
The HTTPConnection.request() and endheaders() both now support chunked encoding request bodies. (Contributed by Demian Brecht and Rolf Krahl in bpo-12319.)

idlelib and IDLE
The idlelib package is being modernized and refactored to make IDLE look and work better and to make the code easier to understand, test, and improve. Part of making IDLE look better, especially on Linux and Mac, is using ttk widgets, mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of either.
‘Modernizing’ includes renaming and consolidation of idlelib modules. The renaming of files with partial uppercase names is similar to the renaming of, for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a result, imports of idlelib files that worked in 3.5 will usually not work in 3.6. At least a module name change will be needed (see idlelib/README.txt), sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in bpo-24225. Most idlelib patches since have been and will be part of the process.)
In compensation, the eventual result with be that some idlelib classes will be easier to use, with better APIs and docstrings explaining them. Additional useful information will be added to idlelib when available.
New in 3.6.2:
Multiple fixes for autocompletion. (Contributed by Louie Lu in bpo-15786.)
New in 3.6.3:
Module Browser (on the File menu, formerly called Class Browser), now displays nested functions and classes in addition to top-level functions and classes. (Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy in bpo-1612262.)
The IDLE features formerly implemented as extensions have been reimplemented as normal features. Their settings have been moved from the Extensions tab to other dialog tabs. (Contributed by Charles Wohlganger and Terry Jan Reedy in bpo-27099.)
The Settings dialog (Options, Configure IDLE) has been partly rewritten to improve both appearance and function. (Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.)
New in 3.6.4:
The font sample now includes a selection of non-Latin characters so that users can better see the effect of selecting a particular font. (Contributed by Terry Jan Reedy in bpo-13802.) The sample can be edited to include other characters. (Contributed by Serhiy Storchaka in bpo-31860.)
New in 3.6.6:
Editor code context option revised. Box displays all context lines up to maxlines. Clicking on a context line jumps the editor to that line. Context colors for custom themes is added to Highlights tab of Settings dialog. (Contributed by Cheryl Sabella and Terry Jan Reedy in bpo-33642, bpo-33768, and bpo-33679.)
On Windows, a new API call tells Windows that tk scales for DPI. On Windows 8.1+ or 10, with DPI compatibility properties of the Python binary unchanged, and a monitor resolution greater than 96 DPI, this should make text and lines sharper. It should otherwise have no effect. (Contributed by Terry Jan Reedy in bpo-33656.)
New in 3.6.7:
Output over N lines (50 by default) is squeezed down to a button. N can be changed in the PyShell section of the General page of the Settings dialog. Fewer, but possibly extra long, lines can be squeezed by right clicking on the output. Squeezed output can be expanded in place by double-clicking the button or into the clipboard or a separate window by right-clicking the button. (Contributed by Tal Einat in bpo-1529353.)

importlib
Import now raises the new exception ModuleNotFoundError (subclass of ImportError) when it cannot find a module. Code that current checks for ImportError (in try-except) will still work. (Contributed by Eric Snow in bpo-15767.)
importlib.util.LazyLoader now calls create_module() on the wrapped loader, removing the restriction that importlib.machinery.BuiltinImporter and importlib.machinery.ExtensionFileLoader couldn’t be used with importlib.util.LazyLoader .
importlib.util.cache_from_source() , importlib.util.source_from_cache() , and importlib.util.spec_from_file_location() now accept a path-like object.

inspect
The inspect.signature() function now reports the implicit .0 parameters generated by the compiler for comprehension and generator expression scopes as if they were positional-only parameters called implicit0 . (Contributed by Jelle Zijlstra in bpo-19611.)
To reduce code churn when upgrading from Python 2.7 and the legacy inspect.getargspec() API, the previously documented deprecation of inspect.getfullargspec() has been reversed. While this function is convenient for single/source Python 2/3 code bases, the richer inspect.signature() interface remains the recommended approach for new code. (Contributed by Nick Coghlan in bpo-27172)

json
json.load() and json.loads() now support binary input. Encoded JSON should be represented using either UTF-8, UTF-16, or UTF-32. (Contributed by Serhiy Storchaka in bpo-17909.)

logging
The new WatchedFileHandler.reopenIfNeeded() method has been added to add the ability to check if the log file needs to be reopened. (Contributed by Marian Horban in bpo-24884.)

math
The tau (τ) constant has been added to the math and cmath modules. (Contributed by Lisa Roach in bpo-12345, see PEP 628 for details.)

multiprocessing
Proxy Objects returned by multiprocessing.Manager() can now be nested. (Contributed by Davin Potts in bpo-6766.)

os
See the summary of PEP 519 for details on how the os and os.path modules now support path-like objects.
scandir() now supports bytes paths on Windows.
A new close() method allows explicitly closing a scandir() iterator. The scandir() iterator now supports the context manager protocol. If a scandir() iterator is neither exhausted nor explicitly closed a ResourceWarning will be emitted in its destructor. (Contributed by Serhiy Storchaka in bpo-25994.)
On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security. See the PEP 524 for the rationale.
The Linux getrandom() syscall (get random bytes) is now exposed as the new os.getrandom() function. (Contributed by Victor Stinner, part of the PEP 524)

pathlib
pathlib now supports path-like objects. (Contributed by Brett Cannon in bpo-27186.)
See the summary of PEP 519 for details.

pdb
The Pdb class constructor has a new optional readrc argument to control whether .pdbrc files should be read.

pickle
Objects that need new called with keyword arguments can now be pickled using pickle protocols older than protocol version 4. Protocol version 4 already supports this case. (Contributed by Serhiy Storchaka in bpo-24164.)

pickletools
pickletools.dis() now outputs the implicit memo index for the MEMOIZE opcode. (Contributed by Serhiy Storchaka in bpo-25382.)

pydoc
The pydoc module has learned to respect the MANPAGER environment variable. (Contributed by Matthias Klose in bpo-8637.)
help() and pydoc can now list named tuple fields in the order they were defined rather than alphabetically. (Contributed by Raymond Hettinger in bpo-24879.)

random
The new choices() function returns a list of elements of specified size from the given population with optional weights. (Contributed by Raymond Hettinger in bpo-18844.)

re
Added support of modifier spans in regular expressions. Examples:
'(?i:p)ython' matches 'python' and 'Python', but not 'PYTHON'; '(?i)g(?-i:v)r' matches 'GvR' and 'gvr', but not 'GVR'. (Contributed by Serhiy Storchaka in bpo-433028.)
Match object groups can be accessed by getitem, which is equivalent to group(). So mo['name'] is now equivalent to mo.group('name'). (Contributed by Eric Smith in bpo-24454.)
Match objects now support index-like objects as group indices. (Contributed by Jeroen Demeyer and Xiang Zhang in bpo-27177.)

readline
Added set_auto_history() to enable or disable automatic addition of input to the history list. (Contributed by Tyler Crompton in bpo-26870.)

rlcompleter
Private and special attribute names now are omitted unless the prefix starts with underscores. A space or a colon is added after some completed keywords. (Contributed by Serhiy Storchaka in bpo-25011 and bpo-25209.)

shlex
The shlex has much improved shell compatibility through the new punctuation_chars argument to control which characters are treated as punctuation. (Contributed by Vinay Sajip in bpo-1521950.)

site
When specifying paths to add to sys.path in a .pth file, you may now specify file paths on top of directories (e.g. zip files). (Contributed by Wolfgang Langner in bpo-26587).

sqlite3
sqlite3.Cursor.lastrowid now supports the REPLACE statement. (Contributed by Alex LordThorsen in bpo-16864.)

socket
The ioctl() function now supports the SIO_LOOPBACK_FAST_PATH control code. (Contributed by Daniel Stokes in bpo-26536.)
The getsockopt() constants SO_DOMAIN , SO_PROTOCOL , SO_PEERSEC , and SO_PASSSEC are now supported. (Contributed by Christian Heimes in bpo-26907.)
The setsockopt() now supports the setsockopt(level, optname, None, optlen: int) form. (Contributed by Christian Heimes in bpo-27744.)
The socket module now supports the address family AF_ALG to interface with Linux Kernel crypto API. ALG_* , SOL_ALG and sendmsg_afalg() were added. (Contributed by Christian Heimes in bpo-27744 with support from Victor Stinner.)
New Linux constants TCP_USER_TIMEOUT and TCP_CONGESTION were added. (Contributed by Omar Sandoval, bpo-26273).

socketserver
Servers based on the socketserver module, including those defined in http.server , xmlrpc.server and wsgiref.simple_server , now support the context manager protocol. (Contributed by Aviv Palivoda in bpo-26404.)
The wfile attribute of StreamRequestHandler classes now implements the io.BufferedIOBase writable interface. In particular, calling write() is now guaranteed to send the data in full. (Contributed by Martin Panter in bpo-26721.)

ssl
ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in bpo-26470.)
3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added. (Contributed by Christian Heimes in bpo-27850 and bpo-27766.)
SSLContext has better default configuration for options and ciphers. (Contributed by Christian Heimes in bpo-28043.)
SSL session can be copied from one client-side connection to another with the new SSLSession class. TLS session resumption can speed up the initial handshake, reduce latency and improve performance (Contributed by Christian Heimes in bpo-19500 based on a draft by Alex Warhawk.)
The new get_ciphers() method can be used to get a list of enabled ciphers in order of cipher priority.
All constants and flags have been converted to IntEnum and IntFlag . (Contributed by Christian Heimes in bpo-28025.)
Server and client-side specific TLS protocols for SSLContext were added. (Contributed by Christian Heimes in bpo-28085.)
Added ssl.SSLContext.post_handshake_auth to enable and ssl.SSLSocket.verify_client_post_handshake() to initiate TLS 1.3 post-handshake authentication. (Contributed by Christian Heimes in gh-78851.)

statistics
A new harmonic_mean() function has been added. (Contributed by Steven D’Aprano in bpo-27181.)

struct
struct now supports IEEE 754 half-precision floats via the 'e' format specifier. (Contributed by Eli Stevens, Mark Dickinson in bpo-11734.)

subprocess
subprocess.Popen destructor now emits a ResourceWarning warning if the child process is still running. Use the context manager protocol ( with proc: ... ) or explicitly call the wait() method to read the exit status of the child process. (Contributed by Victor Stinner in bpo-26741.)
The subprocess.Popen constructor and all functions that pass arguments through to it now accept encoding and errors arguments. Specifying either of these will enable text mode for the stdin , stdout and stderr streams. (Contributed by Steve Dower in bpo-6135.)

sys
The new getfilesystemencodeerrors() function returns the name of the error mode used to convert between Unicode filenames and bytes filenames. (Contributed by Steve Dower in bpo-27781.)
On Windows the return value of the getwindowsversion() function now includes the platform_version field which contains the accurate major version, minor version and build number of the current operating system, rather than the version that is being emulated for the process (Contributed by Steve Dower in bpo-27932.)

telnetlib
telnetlib.Telnet is now a context manager (contributed by Stéphane Wirtel in bpo-25485).

time
The struct_time attributes tm_gmtoff and tm_zone are now available on all platforms.

timeit
The new Timer.autorange() convenience method has been added to call Timer.timeit() repeatedly so that the total run time is greater or equal to 200 milliseconds. (Contributed by Steven D’Aprano in bpo-6422.)
timeit now warns when there is substantial (4x) variance between best and worst times. (Contributed by Serhiy Storchaka in bpo-23552.)

tkinter
Added methods Variable.trace_add() , Variable.trace_remove() , and trace_info() in the tkinter.Variable class. They replace old methods trace_variable() , trace() , trace_vdelete() and trace_vinfo() that use obsolete Tcl commands and might not work in future versions of Tcl. (Contributed by Serhiy Storchaka in bpo-22115).

traceback
Both the traceback module and the interpreter’s builtin exception display now abbreviate long sequences of repeated lines in tracebacks as shown in the following example:

def f():
f()
...

f()
Traceback (most recent call last):
File "", line 1, in
File "", line 1, in f
File "", line 1, in f
File "", line 1, in f
[Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded
(Contributed by Emanuel Barry in bpo-26823.)

tracemalloc
The tracemalloc module now supports tracing memory allocations in multiple different address spaces.
The new DomainFilter filter class has been added to filter block traces by their address space (domain).
(Contributed by Victor Stinner in bpo-26588.)

typing
Since the typing module is provisional, all changes introduced in Python 3.6 have also been backported to Python 3.5.x.
The typing module has a much improved support for generic type aliases. For example Dict[str, Tuple[S, T]] is now a valid type annotation. (Contributed by Guido van Rossum in Github #195.)
The typing.ContextManager class has been added for representing contextlib.AbstractContextManager . (Contributed by Brett Cannon in bpo-25609.)
The typing.Collection class has been added for representing collections.abc.Collection . (Contributed by Ivan Levkivskyi in bpo-27598.)
The typing.ClassVar type construct has been added to mark class variables. As introduced in PEP 526 , a variable annotation wrapped in ClassVar indicates that a given attribute is intended to be used as a class variable and should not be set on instances of that class. (Contributed by Ivan Levkivskyi in Github #280.)
A new TYPE_CHECKING constant that is assumed to be True by the static type checkers, but is False at runtime. (Contributed by Guido van Rossum in Github #230.)
A new NewType() helper function has been added to create lightweight distinct types for annotations:
from typing import NewType
UserId = NewType('UserId', int)
some_id = UserId(524313)
The static type checker will treat the new type as if it were a subclass of the original type. (Contributed by Ivan Levkivskyi in Github #189.)

unicodedata
The unicodedata module now uses data from Unicode 9.0.0. (Contributed by Benjamin Peterson.)

unittest.mock
The Mock class has the following improvements:
• Two new methods, Mock.assert_called() and Mock.assert_called_once() to check if the mock object was called. (Contributed by Amit Saha in bpo-26323.)
• The Mock.reset_mock() method now has two optional keyword only arguments: return_value and side_effect . (Contributed by Kushal Das in bpo-21271.)

urllib.request
If a HTTP request has a file or iterable body (other than a bytes object) but no Content-Length header, rather than throwing an error, AbstractHTTPHandler now falls back to use chunked transfer encoding. (Contributed by Demian Brecht and Rolf Krahl in bpo-12319.)

urllib.robotparser
RobotFileParser now supports the Crawl-delay and Request-rate extensions. (Contributed by Nikolay Bogoychev in bpo-16099.)

venv
venv accepts a new parameter --prompt . This parameter provides an alternative prefix for the virtual environment. (Proposed by Łukasz Balcerzak and ported to 3.6 by Stéphane Wirtel in bpo-22829.)

warnings
A new optional source parameter has been added to the warnings.warn_explicit() function: the destroyed object which emitted a ResourceWarning . A source attribute has also been added to warnings.WarningMessage (contributed by Victor Stinner in bpo-26568 and bpo-26567).
When a ResourceWarning warning is logged, the tracemalloc module is now used to try to retrieve the traceback where the destroyed object was allocated.
Example with the script example.py :
import warnings
def func():
return open(file)
f = func()
f = None
Output of the command python3.6 -Wd -X tracemalloc=5 example.py :
example.py: 7 : ResourceWarning : unclosed file < _io .TextIOWrapper name = 'example.py' mode = 'r' encoding = 'UTF-8' > f = None Object allocated at (most recent call first): File "example.py", lineno 4 return open(file) File "example.py", lineno 6 f = func()
The “Object allocated at” traceback is new and is only displayed if tracemalloc is tracing Python memory allocations and if the warnings module was already imported.

winreg
Added the 64-bit integer type REG_QWORD . (Contributed by Clement Rouault in bpo-23026.)

winsound
Allowed keyword arguments to be passed to Beep , MessageBeep , and PlaySound . (bpo-27982).

xmlrpc.client
The xmlrpc.client module now supports unmarshalling additional data types used by the Apache XML-RPC implementation for numerics and None . (Contributed by Serhiy Storchaka in bpo-26885.)

zipfile
A new ZipInfo.from_file() class method allows making a ZipInfo instance from a filesystem file. A new ZipInfo.is_dir() method can be used to check if the ZipInfo instance represents a directory. (Contributed by Thomas Kluyver in bpo-26039.)
The ZipFile.open() method can now be used to write data into a ZIP file, as well as for extracting data. (Contributed by Thomas Kluyver in bpo-26039.)

zlib
The compress() and decompress() functions now accept keyword arguments. (Contributed by Aviv Palivoda in bpo-26243 and Xiang Zhang in bpo-16764 respectively.)

Optimizations
• The Python interpreter now uses a 16-bit wordcode instead of bytecode which made a number of opcode optimizations possible. (Contributed by Demur Rumed with input and reviews from Serhiy Storchaka and Victor Stinner in bpo-26647 and bpo-28050.)
• The asyncio.Future class now has an optimized C implementation. (Contributed by Yury Selivanov and INADA Naoki in bpo-26081.)
• The asyncio.Task class now has an optimized C implementation. (Contributed by Yury Selivanov in bpo-28544.)
• Various implementation improvements in the typing module (such as caching of generic types) allow up to 30 times performance improvements and reduced memory footprint.
• The ASCII deco

Original source

This article explains the new features in Python 3.9, compared to 3.8. Python 3.9 was released on October 5, 2020. For full details, see the changelog.

See also
PEP 596 - Python 3.9 Release Schedule

Summary – Release highlights
New syntax features:
• PEP 584, union operators added to dict;
• PEP 585, type hinting generics in standard collections;
• PEP 614, relaxed grammar restrictions on decorators.
New built-in features:
• PEP 616, string methods to remove prefixes and suffixes.
New features in the standard library:
• PEP 593, flexible function and variable annotations;
• os.pidfd_open() added that allows process management without races and signals.
Interpreter improvements:
• PEP 573, fast access to module state from methods of C extension types;
• PEP 617, CPython now uses a new parser based on PEG;
• a number of Python builtins (range, tuple, set, frozenset, list, dict) are now sped up using PEP 590 vectorcall;
• garbage collection does not block on resurrected objects;
• a number of Python modules (_abc, audioop, _bz2, _codecs, _contextvars, _crypt, _functools, _json, _locale, math, operator, resource, time, _weakref) now use multiphase initialization as defined by PEP 489;
• a number of standard library modules (audioop, ast, grp, _hashlib, pwd, _posixsubprocess, random, select, struct, termios, zlib) are now using the stable ABI defined by PEP 384.
New library modules:
• PEP 615, the IANA Time Zone Database is now present in the standard library in the zoneinfo module;
• an implementation of a topological sort of a graph is now provided in the new graphlib module.
Release process changes:
• PEP 602, CPython adopts an annual release cycle.

You should check for DeprecationWarning in your code
When Python 2.7 was still supported, a lot of functionality in Python 3 was kept for backward compatibility with Python 2.7. With the end of Python 2 support, these backward compatibility layers have been removed, or will be removed soon. Most of them emitted a DeprecationWarning warning for several years. For example, using collections.Mapping instead of collections.abc.Mapping emits a DeprecationWarning since Python 3.3, released in 2012.
Test your application with the -W default command-line option to see DeprecationWarning and PendingDeprecationWarning, or even with -W error to treat them as errors. Warnings Filter can be used to ignore warnings from third-party code.
Python 3.9 is the last version providing those Python 2 backward compatibility layers, to give more time to Python projects maintainers to organize the removal of the Python 2 support and add support for Python 3.9.
Aliases to Abstract Base Classes in the collections module, like collections.Mapping alias to collections.abc.Mapping, are kept for one last release for backward compatibility. They will be removed from Python 3.10.
More generally, try to run your tests in the Python Development Mode which helps to prepare your code to make it compatible with the next Python version.
Note: a number of pre-existing deprecations were removed in this version of Python as well. Consult the Removed section.

New Features
Dictionary Merge & Update Operators
Merge (|) and update (|=) operators have been added to the built-in dict class. Those complement the existing dict.update and {**d1, **d2} methods of merging dictionaries.
Example:

x = {
"key1": "value1 from x",
"key2": "value2 from x"
}

y = {
"key2": "value2 from y",
"key3": "value3 from y"
}

x | y {'key1': 'value1 from x', 'key2': 'value2 from y', 'key3': 'value3 from y'}
y | x {'key2': 'value2 from x', 'key3': 'value3 from y', 'key1': 'value1 from x'}
See PEP 584 for a full description. (Contributed by Brandt Bucher in bpo-36144.)

New String Methods to Remove Prefixes and Suffixes
str.removeprefix(prefix) and str.removesuffix(suffix) have been added to easily remove an unneeded prefix or a suffix from a string. Corresponding bytes, bytearray, and collections.UserString methods have also been added. See PEP 616 for a full description. (Contributed by Dennis Sweeney in bpo-39939.)

Type Hinting Generics in Standard Collections
In type annotations you can now use built-in collection types such as list and dict as generic types instead of importing the corresponding capitalized types (e.g. List or Dict) from typing. Some other types in the standard library are also now generic, for example queue.Queue.
Example:
def greet_all(names: list[str]) -> None:
for name in names:
print("Hello", name)
See PEP 585 for more details. (Contributed by Guido van Rossum, Ethan Smith, and Batuhan Taşkaya in bpo-39481.)

New Parser
Python 3.9 uses a new parser, based on PEG instead of LL(1). The new parser’s performance is roughly comparable to that of the old parser, but the PEG formalism is more flexible than LL(1) when it comes to designing new language features. We’ll start using this flexibility in Python 3.10 and later.
The ast module uses the new parser and produces the same AST as the old parser.
In Python 3.10, the old parser will be deleted and so will all functionality that depends on it (primarily the parser module, which has long been deprecated). In Python 3.9 only, you can switch back to the LL(1) parser using a command line switch (-X oldparser) or an environment variable (PYTHONOLDPARSER=1).
See PEP 617 for more details. (Contributed by Guido van Rossum, Pablo Galindo and Lysandros Nikolaou in bpo-40334.)

Other Language Changes
• import() now raises ImportError instead of ValueError, which used to occur when a relative import went past its top-level package. (Contributed by Ngalim Siregar in bpo-37444.)
• Python now gets the absolute path of the script filename specified on the command line (ex: python3 script.py): the file attribute of the main module became an absolute path, rather than a relative path. These paths now remain valid after the current directory is changed by os.chdir(). As a side effect, the traceback also displays the absolute path for main module frames in this case. (Contributed by Victor Stinner in bpo-20443.)
• In the Python Development Mode and in debug build, the encoding and errors arguments are now checked for string encoding and decoding operations. Examples: open(), str.encode() and bytes.decode().
• By default, for best performance, the errors argument is only checked at the first encoding/decoding error and the encoding argument is sometimes ignored for empty strings. (Contributed by Victor Stinner in bpo-37388.)
• "".replace("", s, n) now returns s instead of an empty string for all non-zero n. It is now consistent with "".replace("", s). There are similar changes for bytes and bytearray objects. (Contributed by Serhiy Storchaka in bpo-28029.)
• Any valid expression can now be used as a decorator. Previously, the grammar was much more restrictive. See PEP 614 for details. (Contributed by Brandt Bucher in bpo-39702.)
• Improved help for the typing module. Docstrings are now shown for all special forms and special generic aliases (like Union and List). Using help() with generic alias like List[int] will show the help for the correspondent concrete type (list in this case). (Contributed by Serhiy Storchaka in bpo-40257.)
• Parallel running of aclose()/asend()/athrow() is now prohibited, and ag_running now reflects the actual running status of the async generator. (Contributed by Yury Selivanov in bpo-30773.)
• Unexpected errors in calling the iter method are no longer masked by TypeError in the in operator and functions contains(), indexOf() and countOf() of the operator module. (Contributed by Serhiy Storchaka in bpo-40824.)
• Unparenthesized lambda expressions can no longer be the expression part in an if clause in comprehensions and generator expressions. See bpo-41848 and bpo-43755 for details.

New Modules
zoneinfo
The zoneinfo module brings support for the IANA time zone database to the standard library. It adds zoneinfo.ZoneInfo, a concrete datetime.tzinfo implementation backed by the system’s time zone data.
Example:

from zoneinfo import ZoneInfo
import datetime as dt

Daylight saving time

when = dt.datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles"))
print(when)
2020-10-31 12:00:00-07:00

when.tzname()
'PDT'

Standard time

when += dt.timedelta(days=7)
print(when)
2020-11-07 12:00:00-08:00

print(when.tzname())
PST
As a fall-back source of data for platforms that don’t ship the IANA database, the tzdata module was released as a first-party package – distributed via PyPI and maintained by the CPython core team.
See also
PEP 615 – Support for the IANA Time Zone Database in the Standard Library
PEP written and implemented by Paul Ganssle

graphlib
A new module, graphlib, was added that contains the graphlib.TopologicalSorter class to offer functionality to perform topological sorting of graphs. (Contributed by Pablo Galindo, Tim Peters and Larry Hastings in bpo-17005.)

Improved Modules
ast
Added the indent option to dump() which allows it to produce a multiline indented output. (Contributed by Serhiy Storchaka in bpo-37995.)
Added ast.unparse() as a function in the ast module that can be used to unparse an ast.AST object and produce a string with code that would produce an equivalent ast.AST object when parsed. (Contributed by Pablo Galindo and Batuhan Taskaya in bpo-38870.)
Added docstrings to AST nodes that contains the ASDL signature used to construct that node. (Contributed by Batuhan Taskaya in bpo-39638.)

asyncio
Due to significant security concerns, the reuse_address parameter of asyncio.loop.create_datagram_endpoint() is no longer supported. This is because of the behavior of the socket option SO_REUSEADDR in UDP. For more details, see the documentation for loop.create_datagram_endpoint(). (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in bpo-37228.)
Added a new coroutine shutdown_default_executor() that schedules a shutdown for the default executor that waits on the ThreadPoolExecutor to finish closing. Also, asyncio.run() has been updated to use the new coroutine. (Contributed by Kyle Stanley in bpo-34037.)
Added asyncio.PidfdChildWatcher, a Linux-specific child watcher implementation that polls process file descriptors. (bpo-38692)
Added a new coroutine asyncio.to_thread(). It is mainly used for running IO-bound functions in a separate thread to avoid blocking the event loop, and essentially works as a high-level version of run_in_executor() that can directly take keyword arguments. (Contributed by Kyle Stanley and Yury Selivanov in bpo-32309.)
When cancelling the task due to a timeout, asyncio.wait_for() will now wait until the cancellation is complete also in the case when timeout is <= 0, like it does with positive timeouts. (Contributed by Elvis Pranskevichus in bpo-32751.)
asyncio now raises TypeError when calling incompatible methods with an ssl.SSLSocket socket. (Contributed by Ido Michael in bpo-37404.)

compileall
Added new possibility to use hardlinks for duplicated .pyc files: hardlink_dupes parameter and –hardlink-dupes command line option. (Contributed by Lumír ‘Frenzy’ Balhar in bpo-40495.)
Added new options for path manipulation in resulting .pyc files: stripdir, prependdir, limit_sl_dest parameters and -s, -p, -e command line options. Added the possibility to specify the option for an optimization level multiple times. (Contributed by Lumír ‘Frenzy’ Balhar in bpo-38112.)

concurrent.futures
Added a new cancel_futures parameter to concurrent.futures.Executor.shutdown() that cancels all pending futures which have not started running, instead of waiting for them to complete before shutting down the executor. (Contributed by Kyle Stanley in bpo-39349.)
Removed daemon threads from ThreadPoolExecutor and ProcessPoolExecutor. This improves compatibility with subinterpreters and predictability in their shutdown processes. (Contributed by Kyle Stanley in bpo-39812.)
Workers in ProcessPoolExecutor are now spawned on demand, only when there are no available idle workers to reuse. This optimizes startup overhead and reduces the amount of lost CPU time to idle workers. (Contributed by Kyle Stanley in bpo-39207.)

curses
Added curses.get_escdelay(), curses.set_escdelay(), curses.get_tabsize(), and curses.set_tabsize() functions. (Contributed by Anthony Sottile in bpo-38312.)

datetime
The isocalendar() of datetime.date and isocalendar() of datetime.datetime methods now returns a namedtuple() instead of a tuple. (Contributed by Donghee Na in bpo-24416.)

distutils
The upload command now creates SHA2-256 and Blake2b-256 hash digests. It skips MD5 on platforms that block MD5 digest. (Contributed by Christian Heimes in bpo-40698.)

fcntl
Added constants fcntl.F_OFD_GETLK, fcntl.F_OFD_SETLK and fcntl.F_OFD_SETLKW. (Contributed by Donghee Na in bpo-38602.)

ftplib
FTP and FTP_TLS now raise a ValueError if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Donghee Na in bpo-39259.)

gc
When the garbage collector makes a collection in which some objects resurrect (they are reachable from outside the isolated cycles after the finalizers have been executed), do not block the collection of all objects that are still unreachable. (Contributed by Pablo Galindo and Tim Peters in bpo-38379.)
Added a new function gc.is_finalized() to check if an object has been finalized by the garbage collector. (Contributed by Pablo Galindo in bpo-39322.)

hashlib
The hashlib module can now use SHA3 hashes and SHAKE XOF from OpenSSL when available. (Contributed by Christian Heimes in bpo-37630.)
Builtin hash modules can now be disabled with ./configure --without-builtin-hashlib-hashes or selectively enabled with e.g. ./configure --with-builtin-hashlib-hashes=sha3,blake2 to force use of OpenSSL based implementation. (Contributed by Christian Heimes in bpo-40479.)

http
HTTP status codes 103 EARLY_HINTS, 418 IM_A_TEAPOT and 425 TOO_EARLY are added to http.HTTPStatus. (Contributed by Donghee Na in bpo-39509 and Ross Rhodes in bpo-39507.)

IDLE and idlelib
Added option to toggle cursor blink off. (Contributed by Zackery Spytz in bpo-4603.)
Escape key now closes IDLE completion windows. (Contributed by Johnny Najera in bpo-38944.)
Added keywords to module name completion list. (Contributed by Terry J. Reedy in bpo-37765.)
New in 3.9 maintenance releases
Make IDLE invoke sys.excepthook() (when started without ‘-n’). User hooks were previously ignored. (Contributed by Ken Hilton in bpo-43008.)
The changes above have been backported to 3.8 maintenance releases.
Rearrange the settings dialog. Split the General tab into Windows and Shell/Ed tabs. Move help sources, which extend the Help menu, to the Extensions tab. Make space for new options and shorten the dialog. The latter makes the dialog better fit small screens. (Contributed by Terry Jan Reedy in bpo-40468.) Move the indent space setting from the Font tab to the new Windows tab. (Contributed by Mark Roseman and Terry Jan Reedy in bpo-33962.)
Apply syntax highlighting to .pyi files. (Contributed by Alex Waygood and Terry Jan Reedy in bpo-45447.)

imaplib
IMAP4 and IMAP4_SSL now have an optional timeout parameter for their constructors. Also, the open() method now has an optional timeout parameter with this change. The overridden methods of IMAP4_SSL and IMAP4_stream were applied to this change. (Contributed by Donghee Na in bpo-38615.)
imaplib.IMAP4.unselect() is added. imaplib.IMAP4.unselect() frees server’s resources associated with the selected mailbox and returns the server to the authenticated state. This command performs the same actions as imaplib.IMAP4.close(), except that no messages are permanently removed from the currently selected mailbox. (Contributed by Donghee Na in bpo-40375.)

importlib
To improve consistency with import statements, importlib.util.resolve_name() now raises ImportError instead of ValueError for invalid relative import attempts. (Contributed by Ngalim Siregar in bpo-37444.)
Import loaders which publish immutable module objects can now publish immutable packages in addition to individual modules. (Contributed by Dino Viehland in bpo-39336.)
Added importlib.resources.files() function with support for subdirectories in package data, matching backport in importlib_resources version 1.5. (Contributed by Jason R. Coombs in bpo-39791.)
Refreshed importlib.metadata from importlib_metadata version 1.6.1.

inspect
inspect.BoundArguments.arguments is changed from OrderedDict to regular dict. (Contributed by Inada Naoki in bpo-36350 and bpo-39775.)

ipaddress
ipaddress now supports IPv6 Scoped Addresses (IPv6 address with suffix %).
Scoped IPv6 addresses can be parsed using ipaddress.IPv6Address. If present, scope zone ID is available through the scope_id attribute. (Contributed by Oleksandr Pavliuk in bpo-34788.)
Starting with Python 3.9.5 the ipaddress module no longer accepts any leading zeros in IPv4 address strings. (Contributed by Christian Heimes in bpo-36384).

math
Expanded the math.gcd() function to handle multiple arguments. Formerly, it only supported two arguments. (Contributed by Serhiy Storchaka in bpo-39648.)
Added math.lcm(): return the least common multiple of specified arguments. (Contributed by Mark Dickinson, Ananthakrishnan and Serhiy Storchaka in bpo-39479 and bpo-39648.)
Added math.nextafter(): return the next floating-point value after x towards y. (Contributed by Victor Stinner in bpo-39288.)
Added math.ulp(): return the value of the least significant bit of a float. (Contributed by Victor Stinner in bpo-39310.)

multiprocessing
The multiprocessing.SimpleQueue class has a new close() method to explicitly close the queue. (Contributed by Victor Stinner in bpo-30966.)

nntplib
NNTP and NNTP_SSL now raise a ValueError if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Donghee Na in bpo-39259.)

os
Added CLD_KILLED and CLD_STOPPED for si_code. (Contributed by Donghee Na in bpo-38493.)
Exposed the Linux-specific os.pidfd_open() (bpo-38692) and os.P_PIDFD (bpo-38713) for process management with file descriptors.
The os.unsetenv() function is now also available on Windows. (Contributed by Victor Stinner in bpo-39413.)
The os.putenv() and os.unsetenv() functions are now always available. (Contributed by Victor Stinner in bpo-39395.)
Added os.waitstatus_to_exitcode() function: convert a wait status to an exit code. (Contributed by Victor Stinner in bpo-40094.)

pathlib
Added pathlib.Path.readlink() which acts similarly to os.readlink(). (Contributed by Girts Folkmanis in bpo-30618)

pdb
On Windows now Pdb supports ~/.pdbrc. (Contributed by Tim Hopper and Dan Lidral-Porter in bpo-20523.)

poplib
POP3 and POP3_SSL now raise a ValueError if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Donghee Na in bpo-39259.)

pprint
pprint can now pretty-print types.SimpleNamespace. (Contributed by Carl Bordum Hansen in bpo-37376.)

pydoc
The documentation string is now shown not only for class, function, method etc, but for any object that has its own doc attribute. (Contributed by Serhiy Storchaka in bpo-40257.)

random
Added a new random.Random.randbytes() method: generate random bytes. (Contributed by Victor Stinner in bpo-40286.)

signal
Exposed the Linux-specific signal.pidfd_send_signal() for sending to signals to a process using a file descriptor instead of a pid. (bpo-38712)

smtplib
SMTP and SMTP_SSL now raise a ValueError if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Donghee Na in bpo-39259.)
LMTP constructor now has an optional timeout parameter. (Contributed by Donghee Na in bpo-39329.)

socket
The socket module now exports the CAN_RAW_JOIN_FILTERS constant on Linux 4.1 and greater. (Contributed by Stefan Tatschner and Zackery Spytz in bpo-25780.)
The socket module now supports the CAN_J1939 protocol on platforms that support it. (Contributed by Karl Ding in bpo-40291.)
The socket module now has the socket.send_fds() and socket.recv_fds() functions. (Contributed by Joannah Nanjekye, Shinya Okano and Victor Stinner in bpo-28724.)

time
On AIX, thread_time() is now implemented with thread_cputime() which has nanosecond resolution, rather than clock_gettime(CLOCK_THREAD_CPUTIME_ID) which has a resolution of 10 milliseconds. (Contributed by Batuhan Taskaya in bpo-40192)

sys
Added a new sys.platlibdir attribute: name of the platform-specific library directory. It is used to build the path of standard library and the paths of installed extension modules. It is equal to "lib" on most platforms. On Fedora and SuSE, it is equal to "lib64" on 64-bit platforms. (Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in bpo-1294959.)
Previously, sys.stderr was block-buffered when non-interactive. Now stderr defaults to always being line-buffered. (Contributed by Jendrik Seipp in bpo-13601.)

tracemalloc
Added tracemalloc.reset_peak() to set the peak size of traced memory blocks to the current size, to measure the peak of specific pieces of code. (Contributed by Huon Wilson in bpo-40630.)

typing
PEP 593 introduced an typing.Annotated type to decorate existing types with context-specific metadata and new include_extras parameter to typing.get_type_hints() to access the metadata at runtime. (Contributed by Till Varoquaux and Konstantin Kashin.)

unicodedata
The Unicode database has been updated to version 13.0.0. (bpo-39926).

venv
The activation scripts provided by venv now all specify their prompt customization consistently by always using the value specified by VENV_PROMPT. Previously some scripts unconditionally used VENV_PROMPT, others only if it happened to be set (which was the default case), and one used VENV_NAME instead. (Contributed by Brett Cannon in bpo-37663.)

xml
White space characters within attributes are now preserved when serializing xml.etree.ElementTree to XML file. EOLNs are no longer normalized to “n”. This is the result of discussion about how to interpret section 2.11 of XML spec. (Contributed by Mefistotelis in bpo-39011.)

Optimizations
• Optimized the idiom for assignment a temporary variable in comprehensions. Now for y in [expr] in comprehensions is as fast as a simple assignment y = expr . For example:
sums = [s for s in [0] for x in data for s in [s + x]]
Unlike the := operator this idiom does not leak a variable to the outer scope.
(Contributed by Serhiy Storchaka in bpo-32856.)
• Optimized signal handling in multithreaded applications. If a thread different than the main thread gets a signal, the bytecode evaluation loop is no longer interrupted at each bytecode instruction to check for pending signals which cannot be handled. Only the main thread of the main interpreter can handle signals.
Previously, the bytecode evaluation loop was interrupted at each instruction until the main thread handles signals. (Contributed by Victor Stinner in bpo-40010.)
• Optimized the subprocess module on FreeBSD using closefrom(). (Contributed by Ed Maste, Conrad Meyer, Kyle Evans, Kubilay Kocak and Victor Stinner in bpo-38061.)
• PyLong_FromDouble() is now up to 1.87x faster for values that fit into long. (Contributed by Sergey Fedoseev in bpo-37986.)
• A number of Python builtins (range, tuple, set, frozenset, list, dict) are now sped up by using PEP 590 vectorcall protocol. (Contributed by Donghee Na, Mark Shannon, Jeroen Demeyer and Petr Viktorin in bpo-37207.)
• Optimized set.difference_update() for the case when the other set is much larger than the base set. (Suggested by Evgeny Kapun with code contributed by Michele Orrù in bpo-8425.)
• Python’s small object allocator (obmalloc.c) now allows (no more than) one empty arena to remain available for immediate reuse, without returning it to the OS. This prevents thrashing in simple loops where an arena could be created and destroyed anew on each iteration. (Contributed by Tim Peters in bpo-37257.)
• floor division of float operation now has a better performance. Also the message of ZeroDivisionError for this operation is updated. (Contributed by Donghee Na in bpo-39434.)
• Decoding short ASCII strings with UTF-8 and ascii codecs is now about 15% faster. (Contributed by Inada Naoki in bpo-37348.)
Here’s a summary of performance improvements from Python 3.4 through Python 3.9:
Python version 3.4 3.5 3.6 3.7 3.8 3.9

Variable and attribute read access:
read_local 7.1 7.1 5.4 5.1 3.9 3.9
read_nonlocal 7.1 8.1 5.8 5.4 4.4 4.5
read_global 15.5 19.0 14.3 13.6 7.6 7.8
read_builtin 21.1 21.6 18.5 19.0 7.5 7.8
read_classvar_from_class 25.6 26.5 20.7 19.5 18.4 17.9
read_classvar_from_instance 22.8 23.5 18.8 17.1 16.4 16.9
read_instancevar 32.4 33.1 28.0 26.3 25.4 25.3
read_instancevar_slots 27.8 31.3 20.8 20.8 20.2 20.5
read_namedtuple 73.8 57.5 45.0 46.8 18.4 18.7
read_boundmethod 37.6 37.9 29.6 26.9 27.7 41.1

Variable and attribute write access:
write_local 8.7 9.3 5.5 5.3 4.3 4.3
write_nonlocal 10.5 11.1 5.6 5.5 4.7 4.8
write_global 19.7 21.2 18.0 18.0 15.8 16.7
write_classvar 92.9 96.0 104.6 102.1 39.2 39.8
write_instancevar 44.6 45.8 40.0 38.9 35.5 37.4
write_instancevar_slots 35.6 36.1 27.3 26.6 25.7 25.8

Data structure read access:
read_list 24.2 24.5 20.8 20.8 19.0 19.5
read_deque 24.7 25.5 20.2 20.6 19.8 20.2
read_dict 24.3 25.7 22.3 23.0 21.0 22.4
read_strdict 22.6 24.3 19.5 21.2 18.9 21.5

Data structure write access:
write_list 27.1 28.5 22.5 21.6 20.0 20.0
write_deque 28.7 30.1 22.7 21.8 23.5 21.7
write_dict 31.4 33.3 29.3 29.2 24.7 25.4
write_strdict 28.4 29.9 27.5 25.2 23.1 24.5

Stack (or queue) operations:
list_append_pop 93.4 112.7 75.4 74.2 50.8 50.6
deque_append_pop 43.5 57.0 49.4 49.2 42.5 44.2
deque_append_popleft 43.7 57.3 49.7 49.7 42.8 46.4

Timing loop:
loop_overhead 0.5 0.6 0.4 0.3 0.3 0.3
These results were generated from the variable access benchmark script at:
Tools/scripts/var_access_benchmark.py . The benchmark script displays timings in nanoseconds. The benchmarks were measured on an Intel® Core™ i7-4960HQ processor running the macOS 64-bit builds found at python.org.

Deprecated
• The distutils bdist_msi command is now deprecated, use bdist_wheel (wheel packages) instead. (Contributed by Hugo van Kemenade in bpo-39586.)
• Currently math.factorial() accepts float instances with non-negative integer values (like 5.0). It raises a ValueError for non-integral and negative floats. It is now deprecated. In future Python versions it will raise a TypeError for all floats. (Contributed by Serhiy Storchaka in bpo-37315.)
• The parser and symbol modules are deprecated and will be removed in future versions of Python. For the majority of use cases, users can leverage the Abstract Syntax Tree (AST) generation and compilation stage, using the ast module.
• The Public C API functions PyParser_SimpleParseStringFlags(), PyParser_SimpleParseStringFlagsFilename(), PyParser_SimpleParseFileFlags() and PyNode_Compile() are deprecated and will be removed in Python 3.10 together with the old parser.
• Using NotImplemented in a boolean context has been deprecated, as it is almost exclusively the result of incorrect rich comparator implementations. It will be made a TypeError in a future version of Python. (Contributed by Josh Rosenberg in bpo-35712.)
• The random module currently accepts any hashable type as a possible seed value. Unfortunately, some of those types are not guaranteed to have a deterministic hash value. After Python 3.9, the module will restrict its seeds to None, int, float, str, bytes, and bytearray.
• Opening the GzipFile file for writing without specifying the mode argument is deprecated. In future Python versions it will always be opened for reading by default. Specify the mode argument for opening it for writing and silencing a warning. (Contributed by Serhiy Storchaka in bpo-28286.)
• Deprecated the split() method of _tkinter.TkappType in favour of the splitlist() method which has more consistent and predictable behavior. (Contributed by Serhiy Storchaka in bpo-38371.)
• The explicit passing of coroutine objects to asyncio.wait() has been deprecated and will be removed in version 3.11. (Contributed by Yury Selivanov and Kyle Stanley in bpo-34790.)
• binhex4 and hexbin4 standards are now deprecated. The binhex module and the following binascii functions are now deprecated:
• b2a_hqx(), a2b_hqx()
• rlecode_hqx(), rledecode_hqx()
(Contributed by Victor Stinner in bpo-39353.)
• ast classes slice, Index and ExtSlice are considered deprecated and will be removed in future Python versions. value itself should be used instead of Index(value). Tuple(slices, Load()) should be used instead of ExtSlice(slices). (Contributed by Serhiy Storchaka in bpo-34822.)
• ast classes Suite, Param, AugLoad and AugStore are considered deprecated and will be removed in future Python versions. They were not generated by the parser and not accepted by the code generator in Python 3. (Contributed by Batuhan Taskaya in bpo-39639 and bpo-39969 and Serhiy Storchaka in bpo-39988.)
• The PyEval_InitThreads() and PyEval_ThreadsInitialized() functions are now deprecated and will be removed in Python 3.11. Calling PyEval_InitThreads() now does nothing. The GIL is initialized by Py_Initialize() since Python 3.7. (Contributed by Victor Stinner in bpo-39877.)
• Passing None as the first argument to the shlex.split() function has been deprecated. (Contributed by Zackery Spytz in bpo-33262.)
• smtpd.MailmanProxy() is now deprecated as it is unusable without an external module, mailman. (Contributed by Samuel Colvin in bpo-35800.)
• The lib2to3 module now emits a PendingDeprecationWarning. Python 3.9 switched to a PEG parser (see PEP 617), and Python 3.10 may include new language syntax that is not parsable by lib2to3’s LL(1) parser. The lib2to3 module may be removed from the standard library in a future Python version. Consider third-party alternatives such as LibCST or parso. (Contributed by Carl Meyer in bpo-40360.)
• The random parameter of random.shuffle() has been deprecated. (Contributed by Raymond Hettinger in bpo-40465)

Removed
• The erroneous version at unittest.mock.version has been removed.
• nntplib.NNTP: xpath() and xgtitle() methods have been removed. These methods are deprecated since Python 3.3. Generally, these extensions are not supported or not enabled by NNTP server administrators. For xgtitle(), please use nntplib.NNTP.descriptions() or nntplib.NNTP.description() instead. (Contributed by Donghee Na in bpo-39366.)
• array.array: tostring() and fromstring() methods have been removed. They were aliases to tobytes() and frombytes(), deprecated since Python 3.2. (Contributed by Victor Stinner in bpo-38916.)
• The undocumented sys.callstats() function has been removed. Since Python 3.7, it was deprecated and always returned None. It required a special build option CALL_PROFILE which was already removed in Python 3.7. (Contributed by Victor Stinner in bpo-37414.)
• The sys.getcheckinterval() and sys.setcheckinterval() functions have been removed. They were deprecated since Python 3.2. Use sys.getswitchinterval() and sys.setswitchinterval() instead. (Contributed by Victor Stinner in bpo-37392.)
• The C function PyImport_Cleanup() has been removed. It was documented as: “Empty the module table. For internal use only.” (Contributed by Victor Stinner in bpo-36710.)
• _dummy_thread and dummy_threading modules have been removed. These modules were deprecated since Python 3.7 which requires threading support. (Contributed by Victor Stinner in bpo-37312.)
• aifc.openfp() alias to aifc.open(), sunau.openfp() alias to sunau.open(), and wave.openfp() alias to wave.open() have been removed. They were deprecated since Python 3.7. (Contributed by Victor Stinner in bpo-37320.)
• The isAlive() method of threading.Thread has been removed. It was deprecated since Python 3.8. Use is_alive() instead. (Contributed by Donghee Na in bpo-37804.)
• Methods getchildren() and getiterator() of classes ElementTree and Element in the ElementTree module have been removed. They were deprecated in Python 3.2. Use iter(x) or list(x) instead of x.getchildren() and x.iter() or list(x.iter()) instead of x.getiterator(). (Contributed by Serhiy Storchaka in bpo-36543.)
• The old plistlib API has been removed, it was deprecated since Python 3.4. Use the load(), loads(), dump(), and dumps() functions. Additionally, the use_builtin_types parameter was removed, standard bytes objects are always used instead. (Contributed by Jon Janzen in bpo-36409.)
• The C function PyGen_NeedsFinalizing has been removed. It was not documented, tested, or used anywhere within CPython after the implementation of PEP 442. Patch by Joannah Nanjekye. (Contributed by Joannah Nanjekye in bpo-15088)
• base64.encodestring() and base64.decodestring(), aliases deprecated since Python 3.1, have been removed: use base64.encodebytes() and base64.decodebytes() instead. (Contributed by Victor Stinner in bpo-39351.)
• fractions.gcd() function has been removed, it was deprecated since Python 3.5 (bpo-22486): use math.gcd() instead. (Contributed by Victor Stinner in bpo-39350.)
• The buffering parameter of bz2.BZ2File has been removed. Since Python 3.0, it was ignored and using it emitted a DeprecationWarning. Pass an open file object to control how the file is opened. (Contributed by Victor Stinner in bpo-39357.)
• The encoding parameter of json.loads() has been removed. As of Python 3.1, it was deprecated and ignored; using it has emitted a DeprecationWarning since Python 3.8. (Contributed by Inada Naoki in bpo-39377)
• with (await asyncio.lock): and with (yield from asyncio.lock): statements are not longer supported, use async with lock instead. The same is correct for asyncio.Condition and asyncio.Semaphore. (Contributed by Andrew Svetlov in bpo-34793.)
• The sys.getcounts() function, the -X showalloccount command line option and the show_alloc_count field of the C structure PyConfig have been removed. They required a special Python build by defining COUNT_ALLOCS macro. (Contributed by Victor Stinner in bpo-39489.)
• The _field_types attribute of the typing.NamedTuple class has been removed. It was deprecated since Python 3.8. Use the annotations attribute instead. (Contributed by Serhiy Storchaka in bpo-40182.)
• The symtable.SymbolTable.has_exec() method has been removed. It was deprecated since 2006, and only returning False when it’s called. (Contributed by Batuhan Taskaya in bpo-40208)
• The asyncio.Task.current_task() and asyncio.Task.all_tasks() have been removed. They were deprecated since Python 3.7 and you can use asyncio.current_task() and asyncio.all_tasks() instead. (Contributed by Rémi Lapeyre in bpo-40967)
• The unescape() method in the html.parser.HTMLParser class has been removed (it was deprecated since Python 3.4). html.unescape() should be used for converting character references to the corresponding unicode characters.

Porting to Python 3.9
This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python API
• import() and importlib.util.resolve_name() now raise ImportError where it previously raised ValueError. Callers catching the specific exception type and supporting both Python 3.9 and earlier versions will need to catch both using except (ImportError, ValueError):.
• The venv activation scripts no longer special-case when VENV_PROMPT is set to "".
• The select.epoll.unregister() method no longer ignores the EBADF error. (Contributed by Victor Stinner in bpo-39239.)
• The compresslevel parameter of bz2.BZ2File became keyword-only, since the buffering parameter has been removed. (Contributed by Victor Stinner in bpo-39357.)
• Simplified AST for subscription. Simple indices will be represented by their value, extended slices will be represented as tuples. Index(value) will return a value itself, ExtSlice(slices) will return Tuple(slices, Load()). (Contributed by Serhiy Storchaka in bpo-34822.)
• The importlib module now ignores the PYTHONCASEOK environment variable when the -E or -I command line options are being used.
• The encoding parameter has been added to the classes ftplib.FTP and ftplib.FTP_TLS as a keyword-only parameter, and the default encoding is changed from Latin-1 to UTF-8 to follow RFC 2640.
• asyncio.loop.shutdown_default_executor() has been added to AbstractEventLoop, meaning alternative event loops that inherit from it should have this method defined. (Contributed by Kyle Stanley in bpo-34037.)
• The constant values of future flags in the future module is updated in order to prevent collision with compiler flags. Previously PyCF_ALLOW_TOP_LEVEL_AWAIT was clashing with CO_FUTURE_DIVISION. (Contributed by Batuhan Taskaya in bpo-39562.)
• array('u') now uses wchar_t as C type instead of Py_UNICODE. This change doesn’t affect to its behavior because Py_UNICODE is alias of wchar_t since Python 3.3. (Contributed by Inada Naoki in bpo-34538.)
• The logging.getLogger() API now returns the root logger when passed the name 'root', whereas previously it returned a non-root logger named 'root'. This could affect cases where user code explicitly wants a non-root logger named 'root', or instantiates a logger using logging.getLogger(name) in some top-level module called 'root.py'. (Contributed by Vinay Sajip in bpo-37742.)
• Division handling of PurePath now returns NotImplemented instead of raising a TypeError when passed something other than an instance of str or PurePath. This allows creating compatible classes that don’t inherit from those mentioned types. (Contributed by Roger Aiudi in bpo-34775).
• Starting with Python 3.9.5 the ipaddress module no longer accepts any leading zeros in IPv4 address strings. Leading zeros are ambiguous and interpreted as octal notation by some libraries. For example the legacy function socket.inet_aton() treats leading zeros as octal notatation. glibc implementation of modern inet_pton() does not accept any leading zeros. (Contributed by Christian Heimes in bpo-36384).
• codecs.lookup() now normalizes the encoding name the same way as encodings.normalize_encoding(), except that codecs.lookup() also converts the name to lower case. For example, "latex+latin1" encoding name is now normalized to "latex_latin1". (Contributed by Jordon Xu in bpo-37751.)

Changes in the C API
• Instances of heap-allocated types (such as those created with PyType_FromSpec() and similar APIs) hold a reference to their type object since Python 3.8. As indicated in the “Changes in the C API” of Python 3.8, for the vast majority of cases, there should be no side effect but for types that have a custom tp_traverse function, ensure that all custom tp_traverse functions of heap-allocated types visit the object’s type.
Example:
int foo_traverse(PyObject *self, visitproc visit, void *arg)
{
// Rest of the traverse function
#if PY_VERSION_HEX >= 0x03090000
// This was not needed before Python 3.9 (Python issue 35810 and 40217)
Py_VISIT(Py_TYPE(self));
#endif
}
If your traverse function delegates to tp_traverse of its base class (or another type), ensure that Py_TYPE(self) is visited only once. Note that only heap type are expected to visit the type in tp_traverse.
For example, if your tp_traverse function includes:
base->tp_traverse(self, visit, arg)
then add:
#if PY_VERSION_HEX >= 0x03090000
// This was not needed before Python 3.9 (bpo-35810 and bpo-40217)
if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) {
// a heap type's tp_traverse already visited Py_TYPE(self)
}
else
{
Py_VISIT(Py_TYPE(self));
}
#else

(See bpo-35810 and bpo-40217 for more information.)
• The functions PyEval_CallObject, PyEval_CallFunction, PyEval_CallMethod and PyEval_CallObjectWithKeywords are deprecated. Use PyObject_Call() and its variants instead. (See more details in bpo-29548.)

CPython bytecode changes
• The LOAD_ASSERTION_ERROR opcode was added for handling the assert statement. Previously, the assert statement would not work correctly if the AssertionError exception was being shadowed. (Contributed by Zackery Spytz in bpo-34880.)
• The COMPARE_OP opcode was split into four distinct instructions:
• COMPARE_OP for rich comparisons
• IS_OP for ‘is’ and ‘is not’ tests
• CONTAINS_OP for ‘in’ and ‘not in’ tests
• JUMP_IF_NOT_EXC_MATCH for checking exceptions in ‘try-except’ statements.
(Contributed by Mark Shannon in bpo-39156.)

Build Changes
• Added --with-platlibdir option to the configure script: name of the platform-specific library directory, stored in the new sys.platlibdir attribute. See sys.platlibdir attribute for more information. (Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in bpo-1294959.)
• The COUNT_ALLOCS special build macro has been removed. (Contributed by Victor Stinner in bpo-39489.)
• On non-Windows platforms, the setenv() and unsetenv() functions are now required to build Python. (Contributed by Victor Stinner in bpo-39395.)
• On non-Windows platforms, creating bdist_wininst installers is now officially unsupported. (See bpo-10945 for more details.)
• When building Python on macOS from source, _tkinter now links with non-system Tcl and Tk frameworks if they are installed in /Library/Frameworks, as had been the case on older releases of macOS. If a macOS SDK is explicitly configured, by using --enable-universalsdk or -isysroot, only the SDK itself is searched. The default behavior can still be overridden with --with-tcltk-includes and --with-tcltk-libs. (Contributed by Ned Deily in bpo-34956.)
• Python can now be built for Windows 10 ARM64. (Contributed by Steve Dower in bpo-33125.)
• Some individual tests are now skipped when --pgo is used. The tests in question increased the PGO task time significantly and likely didn’t help improve optimization of the final executable. This speeds up the task by a factor of about 15x. Running the full unit test suite is slow. This change may result in a slightly less optimized build since not as many code branches will be executed. If you are willing to wait for the much slower build, the old behavior can be restored using ./configure [..] PROFILE_TASK="-m test --pgo-extended". We make no guarantees as to which PGO task set produces a faster build. Users who care should run their own relevant benchmarks as results can depend on the environment, workload, and compiler tool chain. (See bpo-36044 and bpo-37707 for more details.)

C API Changes
New Features
• PEP 573: Added PyType_FromModuleAndSpec() to associate a module with a class; PyType_GetModule() and PyType_GetModuleState() to retrieve the module and its state; and PyCMethod and METH_METHOD to allow a method to access the class it was defined in. (Contributed by Marcel Plch and Petr Viktorin in bpo-38787.)
• Added PyFrame_GetCode() function: get a frame code. Added PyFrame_GetBack() function: get the frame next outer frame. (Contributed by Victor Stinner in bpo-40421.)
• Added PyFrame_GetLineNumber() to the limited C API. (Contributed by Victor Stinner in bpo-40421.)
• Added PyThreadState_GetInterpreter() and PyInterpreterState_Get() functions to get the interpreter. Added PyThreadState_GetFrame() function to get the current frame of a Python thread state. Added PyThreadState_GetID() function: get the unique identifier of a Python thread state. (Contributed by Victor Stinner in bpo-39947.)
• Added a new public PyObject_CallNoArgs() function to the C API, which calls a callable Python object without any arguments. It is the most efficient way to call a callable Python object without any argument. (Contributed by Victor Stinner in bpo-37194.)
• Changes in the limited C API (if Py_LIMITED_API macro is defined):
• Provide Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() as regular functions for the limited API. Previously, there were defined as macros, but these macros didn’t compile with the limited C API which cannot access PyThreadState.recursion_depth field (the structure is opaque in the limited C API).
• PyObject_INIT() and PyObject_INIT_VAR() become regular “opaque” function to hide implementation details.
(Contributed by Victor Stinner in bpo-38644 and bpo-39542.)
• The PyModule_AddType() function is added to help adding a type to a module. (Contributed by Donghee Na in bpo-40024.)
• Added the functions PyObject_GC_IsTracked() and PyObject_GC_IsFinalized() to the public API to allow to query if Python objects are being currently tracked or have been already finalized by the garbage collector respectively. (Contributed by Pablo Galindo Salgado in bpo-40241.)
• Added _PyObject_FunctionStr() to get a user-friendly string representation of a function-like object. (Patch by Jeroen Demeyer in bpo-37645.)
• Added PyObject_CallOneArg() for calling an object with one positional argument (Patch by Jeroen Demeyer in bpo-37483.)

Porting to Python 3.9
• PyInterpreterState.eval_frame (PEP 523) now requires a new mandatory tstate parameter (PyThreadState*). (Contributed by Victor Stinner in bpo-38500.)
• Extension modules: m_traverse, m_clear and m_free functions of PyModuleDef are no longer called if the module state was requested but is not allocated yet. This is the case immediately after the module is created and before the module is executed (Py_mod_exec function). More precisely, these functions are not called if m_size is greater than 0 and the module state (as returned by PyModule_GetState()) is NULL.
Extension modules without module state (m_size <= 0) are not affected.
• If Py_AddPendingCall() is called in a subinterpreter, the function is now scheduled to be called from the subinterpreter, rather than being called from the main interpreter. Each subinterpreter now has its own list of scheduled calls. (Contributed by Victor Stinner in bpo-39984.)
• The Windows registry is no longer used to initialize sys.path when the -E option is used (if PyConfig.use_environment is set to 0). This is significant when embedding Python on Windows. (Contributed by Zackery Spytz in bpo-8901.)
• The global variable PyStructSequence_UnnamedField is now a constant and refers to a constant string. (Contributed by Serhiy Storchaka in bpo-38650.)
• The PyGC_Head structure is now opaque. It is only defined in the internal C API (pycore_gc.h). (Contributed by Victor Stinner in bpo-40241.)
• The Py_UNICODE_COPY, Py_UNICODE_FILL, PyUnicode_WSTR_LENGTH, PyUnicode_FromUnicode(), PyUnicode_AsUnicode(), _PyUnicode_AsUnicode and PyUnicode_AsUnicodeAndSize() are marked as deprecated in C. They have been deprecated by PEP 393 since Python 3.3. (Contributed by Inada Naoki in bpo-36346.)
• The Py_FatalError() function is replaced with a macro which logs automatically the name of the current function, unless the Py_LIMITED_API macro is defined. (Contributed by Victor Stinner in bpo-39882.)
• The vectorcall protocol now requires that the caller passes only strings as keyword names. (See bpo-37540 for more information.)
• Implementation details of a number of macros and functions are now hidden:
• PyObject_IS_GC() macro was converted to a function.
• The PyObject_NEW() macro becomes an alias to the PyObject_New macro, and the PyObject_NEW_VAR() macro becomes an alias to the PyObject_NewVar macro. They no longer access directly the PyTypeObject.tp_basicsize member.
• PyObject_GET_WEAKREFS_LISTPTR() macro was converted to a function: the macro accessed directly the PyTypeObject.tp_weaklistoffset member.
• PyObject_CheckBuffer() macro was converted to a function: the macro accessed directly the PyTypeObject.tp_as_buffer member.
• PyIndex_Check() is now always declared as an opaque function to hide implementation details: removed the PyIndex_Check() macro. The macro accessed directly the PyTypeObject.tp_as_number member.
(See bpo-40170 for more details.)

Removed
• Excluded PyFPE_START_PROTECT() and PyFPE_END_PROTECT() macros of pyfpe.h from the limited C API. (Contributed by Victor Stinner in bpo-38835.)
• The tp_print slot of PyTypeObject has been removed. It was used for printing objects to files in Python 2.7 and before. Since Python 3.0, it has been ignored and unused. (Contributed by Jeroen Demeyer in bpo-36974.)
• Changes in the limited C API (if Py_LIMITED_API macro is defined):
• Excluded the following functions from the limited C API:
• PyThreadState_DeleteCurrent() (Contributed by Joannah Nanjekye in bpo-37878.)
• _Py_CheckRecursionLimit
• _Py_NewReference()
• _Py_ForgetReference()
• _PyTraceMalloc_NewReference()
• _Py_GetRefTotal()
• The trashcan mechanism which never worked in the limited C A

Original source

This article explains the new features in Python 3.11, compared to 3.10. Python 3.11 was released on October 24, 2022. For full details, see the changelog.

Summary – Release highlights
Python 3.11 is between 10-60% faster than Python 3.10. On average, we measured a 1.25x speedup on the standard benchmark suite. See Faster CPython for details.

New syntax features:
PEP 654: Exception Groups and except*

New built-in features:
PEP 678: Exceptions can be enriched with notes

New standard library modules:
PEP 680: tomllib — Support for parsing TOML in the Standard Library

Interpreter improvements:
PEP 657: Fine-grained error locations in tracebacks
New -P command line option and PYTHONSAFEPATH environment variable to disable automatically prepending potentially unsafe paths to sys.path

New typing features:
PEP 646: Variadic generics
PEP 655: Marking individual TypedDict items as required or not-required
PEP 673: Self type
PEP 675: Arbitrary literal string type
PEP 681: Data class transforms

Important deprecations, removals and restrictions:
PEP 594: Many legacy standard library modules have been deprecated and will be removed in Python 3.13
PEP 624: Py_UNICODE encoder APIs have been removed
PEP 670: Macros converted to static inline functions

New Features
PEP 657: Fine-grained error locations in tracebacks
When printing tracebacks, the interpreter will now point to the exact expression that caused the error, instead of just the line. For example:

Previous versions of the interpreter would point to just the line, making it ambiguous which object was None. These enhanced errors can also be helpful when dealing with deeply nested dict objects and multiple function calls:

As well as complex arithmetic expressions:

Additionally, the information used by the enhanced traceback feature is made available via a general API, that can be used to correlate bytecode instructions with source code location. This information can be retrieved using:
The codeobject.co_positions() method in Python.
The PyCode_Addr2Location() function in the C API.
See PEP 657 for more details. (Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar in bpo-43950.)

Note
This feature requires storing column positions in Code Objects, which may result in a small increase in interpreter memory usage and disk usage for compiled Python files. To avoid storing the extra information and deactivate printing the extra traceback information, use the -X no_debug_ranges command line option or the PYTHONNODEBUGRANGES environment variable.

PEP 654: Exception Groups and except*
PEP 654 introduces language features that enable a program to raise and handle multiple unrelated exceptions simultaneously. The builtin types ExceptionGroup and BaseExceptionGroup make it possible to group exceptions and raise them together, and the new except* syntax generalizes except to match subgroups of exception groups.
See PEP 654 for more details.
(Contributed by Irit Katriel in bpo-45292. PEP written by Irit Katriel, Yury Selivanov and Guido van Rossum.)

PEP 678: Exceptions can be enriched with notes
The add_note() method is added to BaseException. It can be used to enrich exceptions with context information that is not available at the time when the exception is raised. The added notes appear in the default traceback.
See PEP 678 for more details.
(Contributed by Irit Katriel in bpo-45607. PEP written by Zac Hatfield-Dodds.)

Windows py.exe launcher improvements
The copy of the Python install manager included with Python 3.11 has been significantly updated. It now supports company/tag syntax as defined in PEP 514 using the -V:/ argument instead of the limited -.. This allows launching distributions other than PythonCore, the one hosted on python.org.
When using -V: selectors, either company or tag can be omitted, but all installs will be searched. For example, -V:OtherPython/ will select the “best” tag registered for OtherPython, while -V:3.11 or -V:/3.11 will select the “best” distribution with tag 3.11.
When using the legacy -, -., -- or -.- arguments, all existing behaviour should be preserved from past versions, and only releases from PythonCore will be selected. However, the -64 suffix now implies “not 32-bit” (not necessarily x86-64), as there are multiple supported 64-bit platforms. 32-bit runtimes are detected by checking the runtime’s tag for a -32 suffix. All releases of Python since 3.5 have included this in their 32-bit builds.

New Features Related to Type Hints
This section covers major changes affecting PEP 484 type hints and the typing module.

PEP 646: Variadic generics
PEP 484 previously introduced TypeVar, enabling creation of generics parameterised with a single type. PEP 646 adds TypeVarTuple, enabling parameterisation with an arbitrary number of types. In other words, a TypeVarTuple is a variadic type variable, enabling variadic generics.
This enables a wide variety of use cases. In particular, it allows the type of array-like structures in numerical computing libraries such as NumPy and TensorFlow to be parameterised with the array shape. Static type checkers will now be able to catch shape-related bugs in code that uses these libraries.
See PEP 646 for more details.
(Contributed by Matthew Rahtz in bpo-43224, with contributions by Serhiy Storchaka and Jelle Zijlstra. PEP written by Mark Mendoza, Matthew Rahtz, Pradeep Kumar Srinivasan, and Vincent Siles.)

PEP 655: Marking individual TypedDict items as required or not-required
Required and NotRequired provide a straightforward way to mark whether individual items in a TypedDict must be present. Previously, this was only possible using inheritance.
All fields are still required by default, unless the total parameter is set to False, in which case all fields are still not-required by default. For example, the following specifies a TypedDict with one required and one not-required key:

The following definition is equivalent:

See PEP 655 for more details.
(Contributed by David Foster and Jelle Zijlstra in bpo-47087. PEP written by David Foster.)

PEP 673: Self type
The new Self annotation provides a simple and intuitive way to annotate methods that return an instance of their class. This behaves the same as the TypeVar-based approach specified in PEP 484, but is more concise and easier to follow.
Common use cases include alternative constructors provided as classmethod s, and enter() methods that return self:

Self can also be used to annotate method parameters or attributes of the same type as their enclosing class.
See PEP 673 for more details.
(Contributed by James Hilton-Balfe in bpo-46534. PEP written by Pradeep Kumar Srinivasan and James Hilton-Balfe.)

PEP 675: Arbitrary literal string type
The new LiteralString annotation may be used to indicate that a function parameter can be of any literal string type. This allows a function to accept arbitrary literal string types, as well as strings created from other literal strings. Type checkers can then enforce that sensitive functions, such as those that execute SQL statements or shell commands, are called only with static arguments, providing protection against injection attacks.
For example, a SQL query function could be annotated as follows:

See PEP 675 for more details.
(Contributed by Jelle Zijlstra in bpo-47088. PEP written by Pradeep Kumar Srinivasan and Graham Bleaney.)

PEP 681: Data class transforms
@dataclass_transform() may be used to decorate a class, metaclass, or a function that is itself a decorator. The presence of @dataclass_transform() tells a static type checker that the decorated object performs runtime “magic” that transforms a class, giving it dataclass-like behaviors.
For example:

See PEP 681 for more details.
(Contributed by Jelle Zijlstra in gh-91860. PEP written by Erik De Bonte and Eric Traut.)

PEP 563 may not be the future
PEP 563 Postponed Evaluation of Annotations (the from future import annotations future statement) that was originally planned for release in Python 3.10 has been put on hold indefinitely. See this message from the Steering Council for more information.

Other Language Changes
Starred unpacking expressions can now be used in for statements. (See bpo-46725 for more details.)
Asynchronous comprehensions are now allowed inside comprehensions in asynchronous functions. Outer comprehensions implicitly become asynchronous in this case. (Contributed by Serhiy Storchaka in bpo-33346.)
A TypeError is now raised instead of an AttributeError in with statements and contextlib.ExitStack.enter_context() for objects that do not support the context manager protocol, and in async with statements and contextlib.AsyncExitStack.enter_async_context() for objects not supporting the asynchronous context manager protocol. (Contributed by Serhiy Storchaka in bpo-12022 and bpo-44471.)
Added object.getstate(), which provides the default implementation of the getstate() method. copy ing and pickle ing instances of subclasses of builtin types bytearray, set, frozenset, collections.OrderedDict, collections.deque, weakref.WeakSet, and datetime.tzinfo now copies and pickles instance attributes implemented as slots. This change has an unintended side effect: It trips up a small minority of existing Python projects not expecting object.getstate() to exist. See the later comments on gh-70766 for discussions of what workarounds such code may need. (Contributed by Serhiy Storchaka in bpo-26579.)
Added a -P command line option and a PYTHONSAFEPATH environment variable, which disable the automatic prepending to sys.path of the script’s directory when running a script, or the current directory when using -c and -m . This ensures only stdlib and installed modules are picked up by import , and avoids unintentionally or maliciously shadowing modules with those in a local (and typically user-writable) directory. (Contributed by Victor Stinner in gh-57684.)
A "z" option was added to the Format specification mini-language that coerces negative to positive zero after rounding to the format precision. See PEP 682 for more details. (Contributed by John Belmonte in gh-90153.)
Bytes are no longer accepted on sys.path . Support broke sometime between Python 3.2 and 3.6, with no one noticing until after Python 3.10.0 was released. In addition, bringing back support would be problematic due to interactions between -b and sys.path_importer_cache when there is a mixture of str and bytes keys. (Contributed by Thomas Grainger in gh-91181.)

Other CPython Implementation Changes
The special methods complex() for complex and bytes() for bytes are implemented to support the typing.SupportsComplex and typing.SupportsBytes protocols. (Contributed by Mark Dickinson and Donghee Na in bpo-24234.)
siphash13 is added as a new internal hashing algorithm. It has similar security properties as siphash24 , but it is slightly faster for long inputs. str , bytes , and some other types now use it as the default algorithm for hash() . PEP 552 hash-based .pyc files now use siphash13 too. (Contributed by Inada Naoki in bpo-29410.)
When an active exception is re-raised by a raise statement with no parameters, the traceback attached to this exception is now always sys.exc_info()[1].traceback . This means that changes made to the traceback in the current except clause are reflected in the re-raised exception. (Contributed by Irit Katriel in bpo-45711.)
The interpreter state’s representation of handled exceptions (aka exc_info or _PyErr_StackItem ) now only has the exc_value field; exc_type and exc_traceback have been removed, as they can be derived from exc_value . (Contributed by Irit Katriel in bpo-45711.)
A new command line option, AppendPath , has been added for the Windows installer. It behaves similarly to PrependPath , but appends the install and scripts directories instead of prepending them. (Contributed by Bastian Neuburger in bpo-44934.)
The PyConfig.module_search_paths_set field must now be set to 1 for initialization to use PyConfig.module_search_paths to initialize sys.path . Otherwise, initialization will recalculate the path and replace any values added to module_search_paths .
The output of the --help option now fits in 50 lines/80 columns. Information about Python environment variables and -X options is now available using the respective --help-env and --help-xoptions flags, and with the new --help-all . (Contributed by Éric Araujo in bpo-46142.)
Converting between int and str in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a ValueError if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE 2020-10735 . This limit can be configured or disabled by environment variable, command line flag, or sys APIs. See the integer string conversion length limitation documentation. The default limit is 4300 digits in string form.

New Modules
tomllib : For parsing TOML. See PEP 680 for more details. (Contributed by Taneli Hukkinen in bpo-40059.)
wsgiref.types : WSGI-specific types for static type checking. (Contributed by Sebastian Rittau in bpo-42012.)

Improved Modules
asyncio
Added the TaskGroup class, an asynchronous context manager holding a group of tasks that will wait for all of them upon exit. For new code this is recommended over using create_task() and gather() directly. (Contributed by Yury Selivanov and others in gh-90908.)
Added timeout() , an asynchronous context manager for setting a timeout on asynchronous operations. For new code this is recommended over using wait_for() directly. (Contributed by Andrew Svetlov in gh-90927.)
Added the Runner class, which exposes the machinery used by run() . (Contributed by Andrew Svetlov in gh-91218.)
Added the Barrier class to the synchronization primitives in the asyncio library, and the related BrokenBarrierError exception. (Contributed by Yves Duprat and Andrew Svetlov in gh-87518.)
Added keyword argument all_errors to asyncio.loop.create_connection() so that multiple connection errors can be raised as an ExceptionGroup .
Added the asyncio.StreamWriter.start_tls() method for upgrading existing stream-based connections to TLS. (Contributed by Ian Good in bpo-34975.)
Added raw datagram socket functions to the event loop: sock_sendto() , sock_recvfrom() and sock_recvfrom_into() . These have implementations in SelectorEventLoop and ProactorEventLoop . (Contributed by Alex Grönholm in bpo-46805.)
Added cancelling() and uncancel() methods to Task . These are primarily intended for internal use, notably by TaskGroup .

contextlib
Added non parallel-safe chdir() context manager to change the current working directory and then restore it on exit. Simple wrapper around chdir() . (Contributed by Filipe Laíns in bpo-25625)

dataclasses
Change field default mutability check, allowing only defaults which are hashable instead of any object which is not an instance of dict , list or set . (Contributed by Eric V. Smith in bpo-44674.)

datetime
Add datetime.UTC , a convenience alias for datetime.timezone.utc . (Contributed by Kabir Kwatra in gh-91973.)
datetime.date.fromisoformat() , datetime.time.fromisoformat() and datetime.datetime.fromisoformat() can now be used to parse most ISO 8601 formats (barring only those that support fractional hours and minutes). (Contributed by Paul Ganssle in gh-80010.)

enum
Renamed EnumMeta to EnumType (EnumMeta kept as an alias).
Added StrEnum , with members that can be used as (and must be) strings.
Added ReprEnum , which only modifies the repr() of members while returning their literal values (rather than names) for str() and format() (used by str() , format() and f-strings).
Changed Enum.format() (the default for format() , str.format() and f-strings) to always produce the same result as Enum.str() : for enums inheriting from ReprEnum it will be the member’s value; for all other enums it will be the enum and member name (e.g. Color.RED ).
Added a new boundary class parameter to Flag enums and the FlagBoundary enum with its options, to control how to handle out-of-range flag values.
Added the verify() enum decorator and the EnumCheck enum with its options, to check enum classes against several specific constraints.
Added the member() and nonmember() decorators, to ensure the decorated object is/is not converted to an enum member.
Added the property() decorator, which works like property() except for enums. Use this instead of types.DynamicClassAttribute() .
Added the global_enum() enum decorator, which adjusts repr() and str() to show values as members of their module rather than the enum class. For example, 're.ASCII' for the ASCII member of re.RegexFlag rather than 'RegexFlag.ASCII' .
Enhanced Flag to support len() , iteration and in / not in on its members. For example, the following now works: len(AFlag(3)) == 2 and list(AFlag(3)) == (AFlag.ONE, AFlag.TWO)
Changed Enum and Flag so that members are now defined before init_subclass() is called; dir() now includes methods, etc., from mixed-in data types.
Changed Flag to only consider primary values (power of two) canonical while composite values ( 3 , 6 , 10 , etc.) are considered aliases; inverted flags are coerced to their positive equivalent.

fcntl
On FreeBSD, the F_DUP2FD and F_DUP2FD_CLOEXEC flags respectively are supported, the former equals to dup2 usage while the latter set the FD_CLOEXEC flag in addition.

fractions
Support PEP 515 -style initialization of Fraction from string. (Contributed by Sergey B Kirpichev in bpo-44258.)
Fraction now implements an int method, so that an isinstance(some_fraction, typing.SupportsInt) check passes. (Contributed by Mark Dickinson in bpo-44547.)

functools
functools.singledispatch() now supports types.UnionType and typing.Union as annotations to the dispatch argument.

gzip
The gzip.compress() function is now faster when used with the mtime=0 argument as it delegates the compression entirely to a single zlib.compress() operation. There is one side effect of this change: The gzip file header contains an “OS” byte in its header. That was traditionally always set to a value of 255 representing “unknown” by the gzip module. Now, when using compress() with mtime=0 , it may be set to a different value by the underlying zlib C library Python was linked against. (See gh-112346 for details on the side effect.)

hashlib
hashlib.blake2b() and hashlib.blake2s() now prefer libb2 over Python’s vendored copy. (Contributed by Christian Heimes in bpo-47095.)
The internal _sha3 module with SHA3 and SHAKE algorithms now uses tiny_sha3 instead of the Keccak Code Package to reduce code and binary size. The hashlib module prefers optimized SHA3 and SHAKE implementations from OpenSSL. The change affects only installations without OpenSSL support. (Contributed by Christian Heimes in bpo-47098.)
Add hashlib.file_digest() , a helper function for efficient hashing of files or file-like objects. (Contributed by Christian Heimes in gh-89313.)

IDLE and idlelib
Apply syntax highlighting to .pyi files. (Contributed by Alex Waygood and Terry Jan Reedy in bpo-45447.)
Include prompts when saving Shell with inputs and outputs. (Contributed by Terry Jan Reedy in gh-95191.)

inspect
Add getmembers_static() to return all members without triggering dynamic lookup via the descriptor protocol. (Contributed by Weipeng Hong in bpo-30533.)
Add ismethodwrapper() for checking if the type of an object is a MethodWrapperType . (Contributed by Hakan Çelik in bpo-29418.)
Change the frame-related functions in the inspect module to return new FrameInfo and Traceback class instances (backwards compatible with the previous named tuple-like interfaces) that includes the extended PEP 657 position information (end line number, column and end column). The affected functions are:
inspect.getframeinfo()
inspect.getouterframes()
inspect.getinnerframes()
inspect.stack()
inspect.trace()
(Contributed by Pablo Galindo in gh-88116.)

locale
Add locale.getencoding() to get the current locale encoding. It is similar to locale.getpreferredencoding(False) but ignores the Python UTF-8 Mode .

logging
Added getLevelNamesMapping() to return a mapping from logging level names (e.g. 'CRITICAL' ) to the values of their corresponding Logging Levels (e.g. 50 , by default). (Contributed by Andrei Kulakovin in gh-88024.)
Added a createSocket() method to SysLogHandler , to match SocketHandler.createSocket() . It is called automatically during handler initialization and when emitting an event, if there is no active socket. (Contributed by Kirill Pinchuk in gh-88457.)

math
Add math.exp2() : return 2 raised to the power of x. (Contributed by Gideon Mitchell in bpo-45917.)
Add math.cbrt() : return the cube root of x. (Contributed by Ajith Ramachandran in bpo-44357.)
The behaviour of two math.pow() corner cases was changed, for consistency with the IEEE 754 specification. The operations math.pow(0.0, -math.inf) and math.pow(-0.0, -math.inf) now return inf . Previously they raised ValueError . (Contributed by Mark Dickinson in bpo-44339.)
The math.nan value is now always available. (Contributed by Victor Stinner in bpo-46917.)

operator
A new function operator.call has been added, such that operator.call(obj, *args, **kwargs) == obj(*args, **kwargs) . (Contributed by Antony Lee in bpo-44019.)

os
On Windows, os.urandom() now uses BCryptGenRandom() , instead of CryptGenRandom() which is deprecated. (Contributed by Donghee Na in bpo-44611.)

pathlib
glob() and rglob() return only directories if pattern ends with a pathname components separator: sep or altsep . (Contributed by Eisuke Kawasima in bpo-22276 and bpo-33392.)

re
Atomic grouping ((?>...) ) and possessive quantifiers ( *+ , ++ , ?+ , {m,n}+ ) are now supported in regular expressions. (Contributed by Jeffrey C. Jacobs and Serhiy Storchaka in bpo-433030.)

shutil
Add optional parameter dir_fd in shutil.rmtree() . (Contributed by Serhiy Storchaka in bpo-46245.)

socket
Add CAN Socket support for NetBSD. (Contributed by Thomas Klausner in bpo-30512.)
create_connection() has an option to raise, in case of failure to connect, an ExceptionGroup containing all errors instead of only raising the last error. (Contributed by Irit Katriel in bpo-29980.)

sqlite3
You can now disable the authorizer by passing None to set_authorizer() . (Contributed by Erlend E. Aasland in bpo-44491.)
Collation name create_collation() can now contain any Unicode character. Collation names with invalid characters now raise UnicodeEncodeError instead of sqlite3.ProgrammingError . (Contributed by Erlend E. Aasland in bpo-44688.)
sqlite3 exceptions now include the SQLite extended error code as sqlite_errorcode and the SQLite error name as sqlite_errorname . (Contributed by Aviv Palivoda, Daniel Shahaf, and Erlend E. Aasland in bpo-16379 and bpo-24139.)
Add setlimit() and getlimit() to sqlite3.Connection for setting and getting SQLite limits by connection basis. (Contributed by Erlend E. Aasland in bpo-45243.)
sqlite3 now sets sqlite3.threadsafety based on the default threading mode the underlying SQLite library has been compiled with. (Contributed by Erlend E. Aasland in bpo-45613.)
sqlite3 C callbacks now use unraisable exceptions if callback tracebacks are enabled. Users can now register an unraisable hook handler to improve their debug experience. (Contributed by Erlend E. Aasland in bpo-45828.)
Fetch across rollback no longer raises InterfaceError . Instead we leave it to the SQLite library to handle these cases. (Contributed by Erlend E. Aasland in bpo-44092.)
Add serialize() and deserialize() to sqlite3.Connection for serializing and deserializing databases. (Contributed by Erlend E. Aasland in bpo-41930.)
Add create_window_function() to sqlite3.Connection for creating aggregate window functions. (Contributed by Erlend E. Aasland in bpo-34916.)
Add blobopen() to sqlite3.Connection . sqlite3.Blob allows incremental I/O operations on blobs. (Contributed by Aviv Palivoda and Erlend E. Aasland in bpo-24905.)

string
Add get_identifiers() and is_valid() to string.Template , which respectively return all valid placeholders, and whether any invalid placeholders are present. (Contributed by Ben Kehoe in gh-90465.)

sys
sys.exc_info() now derives the type and traceback fields from the value (the exception instance), so when an exception is modified while it is being handled, the changes are reflected in the results of subsequent calls to exc_info() . (Contributed by Irit Katriel in bpo-45711.)
Add sys.exception() which returns the active exception instance (equivalent to sys.exc_info()[1] ). (Contributed by Irit Katriel in bpo-46328.)
Add the sys.flags.safe_path flag. (Contributed by Victor Stinner in gh-57684.)

sysconfig
Three new installation schemes (posix_venv , nt_venv and venv ) were added and are used when Python creates new virtual environments or when it is running from a virtual environment. The first two schemes ( posix_venv and nt_venv ) are OS-specific for non-Windows and Windows, the venv is essentially an alias to one of them according to the OS Python runs on. This is useful for downstream distributors who modify sysconfig.get_preferred_scheme() . Third party code that creates new virtual environments should use the new venv installation scheme to determine the paths, as does venv . (Contributed by Miro Hrončok in bpo-45413.)

tempfile
SpooledTemporaryFile objects now fully implement the methods of io.BufferedIOBase or io.TextIOBase (depending on file mode). This lets them work correctly with APIs that expect file-like objects, such as compression modules. (Contributed by Carey Metcalfe in gh-70363.)

threading
On Unix, if the sem_clockwait() function is available in the C library (glibc 2.30 and newer), the threading.Lock.acquire() method now uses the monotonic clock ( time.CLOCK_MONOTONIC ) for the timeout, rather than using the system clock ( time.CLOCK_REALTIME ), to not be affected by system clock changes. (Contributed by Victor Stinner in bpo-41710.)

time
On Unix, time.sleep() now uses the clock_nanosleep() or nanosleep() function, if available, which has a resolution of 1 nanosecond (10^-9 seconds), rather than using select() which has a resolution of 1 microsecond (10^-6 seconds). (Contributed by Benjamin Szőke and Victor Stinner in bpo-21302.)
On Windows 8.1 and newer, time.sleep() now uses a waitable timer based on high-resolution timers which has a resolution of 100 nanoseconds (10^-7 seconds). Previously, it had a resolution of 1 millisecond (10^-3 seconds). (Contributed by Benjamin Szőke, Donghee Na, Eryk Sun and Victor Stinner in bpo-21302 and bpo-45429.)

tkinter
Added method info_patchlevel() which returns the exact version of the Tcl library as a named tuple similar to sys.version_info . (Contributed by Serhiy Storchaka in gh-91827.)

traceback
Add traceback.StackSummary.format_frame_summary() to allow users to override which frames appear in the traceback, and how they are formatted. (Contributed by Ammar Askar in bpo-44569.)
Add traceback.TracebackException.print() , which prints the formatted TracebackException instance to a file. (Contributed by Irit Katriel in bpo-33809.)

typing
For major changes, see New Features Related to Type Hints.
Add typing.assert_never() and typing.Never . typing.assert_never() is useful for asking a type checker to confirm that a line of code is not reachable. At runtime, it raises an AssertionError . (Contributed by Jelle Zijlstra in gh-90633.)
Add typing.reveal_type() . This is useful for asking a type checker what type it has inferred for a given expression. At runtime it prints the type of the received value. (Contributed by Jelle Zijlstra in gh-90572.)
Add typing.assert_type() . This is useful for asking a type checker to confirm that the type it has inferred for a given expression matches the given type. At runtime it simply returns the received value. (Contributed by Jelle Zijlstra in gh-90638.)
typing.TypedDict types can now be generic. (Contributed by Samodya Abeysiriwardane in gh-89026.)
NamedTuple types can now be generic. (Contributed by Serhiy Storchaka in bpo-43923.)
Allow subclassing of typing.Any . This is useful for avoiding type checker errors related to highly dynamic class, such as mocks. (Contributed by Shantanu Jain in gh-91154.)
The typing.final() decorator now sets the final attributed on the decorated object. (Contributed by Jelle Zijlstra in gh-90500.)
The typing.get_overloads() function can be used for introspecting the overloads of a function. typing.clear_overloads() can be used to clear all registered overloads of a function. (Contributed by Jelle Zijlstra in gh-89263.)
The init() method of Protocol subclasses is now preserved. (Contributed by Adrian Garcia Badarasco in gh-88970.)
The representation of empty tuple types ( Tuple[()] ) is simplified. This affects introspection, e.g. get_args(Tuple[()]) now evaluates to () instead of ((),) . (Contributed by Serhiy Storchaka in gh-91137.)
Loosen runtime requirements for type annotations by removing the callable check in the private typing._type_check function. (Contributed by Gregory Beauregard in gh-90802.)
typing.get_type_hints() now supports evaluating strings as forward references in PEP 585 generic aliases . (Contributed by Niklas Rosenstein in gh-85542.)
typing.get_type_hints() no longer adds Optional to parameters with None as a default. (Contributed by Nikita Sobolev in gh-90353.)
typing.get_type_hints() now supports evaluating bare stringified ClassVar annotations. (Contributed by Gregory Beauregard in gh-90711.)
typing.no_type_check() no longer modifies external classes and functions. It also now correctly marks classmethods as not to be type checked. (Contributed by Nikita Sobolev in gh-90729.)

unicodedata
The Unicode database has been updated to version 14.0.0. (Contributed by Benjamin Peterson in bpo-45190).

unittest
Added methods enterContext() and enterClassContext() of class TestCase , method enterAsyncContext() of class IsolatedAsyncioTestCase and function unittest.enterModuleContext() . (Contributed by Serhiy Storchaka in bpo-45046.)

venv
When new Python virtual environments are created, the venv installation scheme is used to determine the paths inside the environment. When Python runs in a virtual environment, the same installation scheme is the default. That means that downstream distributors can change the default sysconfig install scheme without changing behavior of virtual environments. Third party code that also creates new virtual environments should do the same. (Contributed by Miro Hrončok in bpo-45413.)

warnings
warnings.catch_warnings() now accepts arguments for warnings.simplefilter() , providing a more concise way to locally ignore warnings or convert them to errors. (Contributed by Zac Hatfield-Dodds in bpo-47074.)

zipfile
Added support for specifying member name encoding for reading metadata in a ZipFile’s directory and file headers. (Contributed by Stephen J. Turnbull and Serhiy Storchaka in bpo-28080.)
Added ZipFile.mkdir() for creating new directories inside ZIP archives. (Contributed by Sam Ezeh in gh-49083.)
Added stem , suffix and suffixes to zipfile.Path . (Contributed by Miguel Brito in gh-88261.)

Optimizations
This section covers specific optimizations independent of the Faster CPython project, which is covered in its own section.
The compiler now optimizes simple printf-style % formatting on string literals containing only the format codes %s , %r and %a and makes it as fast as a corresponding f-string expression. (Contributed by Serhiy Storchaka in bpo-28307.)
Integer division ( // ) is better tuned for optimization by compilers. It is now around 20% faster on x86-64 when dividing an int by a value smaller than 230 . (Contributed by Gregory P. Smith and Tim Peters in gh-90564.)
sum() is now nearly 30% faster for integers smaller than 230 . (Contributed by Stefan Behnel in gh-68264.)
Resizing lists is streamlined for the common case, speeding up list.append() by ≈15% and simple list comprehension s by up to 20-30% (Contributed by Dennis Sweeney in gh-91165.)
Dictionaries don’t store hash values when all keys are Unicode objects, decreasing dict size. For example, sys.getsizeof(dict.fromkeys("abcdefg")) is reduced from 352 bytes to 272 bytes (23% smaller) on 64-bit platforms. (Contributed by Inada Naoki in bpo-46845.)
Using asyncio.DatagramProtocol is now orders of magnitude faster when transferring large files over UDP, with speeds over 100 times higher for a ≈60 MiB file. (Contributed by msoxzw in gh-91487.)
math functions comb() and perm() are now ≈10 times faster for large arguments (with a larger speedup for larger k ). (Contributed by Serhiy Storchaka in bpo-37295.)
The statistics functions mean() , variance() and stdev() now consume iterators in one pass rather than converting them to a list first. This is twice as fast and can save substantial memory. (Contributed by Raymond Hettinger in gh-90415.)
unicodedata.normalize() now normalizes pure-ASCII strings in constant time. (Contributed by Donghee Na in bpo-44987.)

Faster CPython
CPython 3.11 is an average of 25% faster than CPython 3.10 as measured with the pyperformance benchmark suite, when compiled with GCC on Ubuntu Linux. Depending on your workload, the overall speedup could be 10-60%.
This project focuses on two major areas in Python: Faster Startup and Faster Runtime. Optimizations not covered by this project are listed separately under Optimizations .

Faster Startup
Frozen imports / Static code objects
Python caches bytecode in the pycache directory to speed up module loading.
Previously in 3.10, Python module execution looked like this:
Read pycache -> Unmarshal -> Heap allocated code object -> Evaluate
In Python 3.11, the core modules essential for Python startup are “frozen”. This means that their Code Objects (and bytecode) are statically allocated by the interpreter. This reduces the steps in module execution process to:
Statically allocated code object -> Evaluate
Interpreter startup is now 10-15% faster in Python 3.11. This has a big impact for short-running programs using Python.
(Contributed by Eric Snow, Guido van Rossum and Kumar Aditya in many issues.)

Faster Runtime
Cheaper, lazy Python frames
Python frames, holding execution information, are created whenever Python calls a Python function. The following are new frame optimizations:
Streamlined the frame creation process.
Avoided memory allocation by generously re-using frame space on the C stack.
Streamlined the internal frame struct to contain only essential information. Frames previously held extra debugging and memory management information.
Old-style frame objects are now created only when requested by debuggers or by Python introspection functions such as sys._getframe() and inspect.currentframe() . For most user code, no frame objects are created at all. As a result, nearly all Python functions calls have sped up significantly. We measured a 3-7% speedup in pyperformance.
(Contributed by Mark Shannon in bpo-44590.)

Inlined Python function calls
During a Python function call, Python will call an evaluating C function to interpret that function’s code. This effectively limits pure Python recursion to what’s safe for the C stack.
In 3.11, when CPython detects Python code calling another Python function, it sets up a new frame, and “jumps” to the new code inside the new frame. This avoids calling the C interpreting function altogether.
Most Python function calls now consume no C stack space, speeding them up. In simple recursive functions like fibonacci or factorial, we observed a 1.7x speedup. This also means recursive functions can recurse significantly deeper (if the user increases the recursion limit with sys.setrecursionlimit() ). We measured a 1-3% improvement in pyperformance.
(Contributed by Pablo Galindo and Mark Shannon in bpo-45256.)

PEP 659: Specializing Adaptive Interpreter
PEP 659 is one of the key parts of the Faster CPython project. The general idea is that while Python is a dynamic language, most code has regions where objects and types rarely change. This concept is known as type stability .
At runtime, Python will try to look for common patterns and type stability in the executing code. Python will then replace the current operation with a more specialized one. This specialized operation uses fast paths available only to those use cases/types, which generally outperform their generic counterparts. This also brings in another concept called inline caching , where Python caches the results of expensive operations directly in the bytecode .
The specializer will also combine certain common instruction pairs into one superinstruction, reducing the overhead during execution.
Python will only specialize when it sees code that is “hot” (executed multiple times). This prevents Python from wasting time on run-once code. Python can also de-specialize when code is too dynamic or when the use changes. Specialization is attempted periodically, and specialization attempts are not too expensive, allowing specialization to adapt to new circumstances.
(PEP written by Mark Shannon, with ideas inspired by Stefan Brunthaler. See PEP 659 for more information. Implementation by Mark Shannon and Brandt Bucher, with additional help from Irit Katriel and Dennis Sweeney.)

Misc
Objects now require less memory due to lazily created object namespaces. Their namespace dictionaries now also share keys more freely. (Contributed Mark Shannon in bpo-45340 and bpo-40116.)
“Zero-cost” exceptions are implemented, eliminating the cost of try statements when no exception is raised. (Contributed by Mark Shannon in bpo-40222.)
A more concise representation of exceptions in the interpreter reduced the time required for catching an exception by about 10%. (Contributed by Irit Katriel in bpo-45711.)
re ’s regular expression matching engine has been partially refactored, and now uses computed gotos (or “threaded code”) on supported platforms. As a result, Python 3.11 executes the pyperformance regular expression benchmarks up to 10% faster than Python 3.10. (Contributed by Brandt Bucher in gh-91404.)

FAQ
How should I write my code to utilize these speedups?
Write Pythonic code that follows common best practices; you don’t have to change your code. The Faster CPython project optimizes for common code patterns we observe.
Will CPython 3.11 use more memory?
Maybe not; we don’t expect memory use to exceed 20% higher than 3.10. This is offset by memory optimizations for frame objects and object dictionaries as mentioned above.
I don’t see any speedups in my workload. Why?
Certain code won’t have noticeable benefits. If your code spends most of its time on I/O operations, or already does most of its computation in a C extension library like NumPy, there won’t be significant speedups. This project currently benefits pure-Python workloads the most.
Furthermore, the pyperformance figures are a geometric mean. Even within the pyperformance benchmarks, certain benchmarks have slowed down slightly, while others have sped up by nearly 2x!
Is there a JIT compiler?
No. We’re still exploring other optimizations.

About
Faster CPython explores optimizations for CPython . The main team is funded by Microsoft to work on this full-time. Pablo Galindo Salgado is also funded by Bloomberg LP to work on the project part-time. Finally, many contributors are volunteers from the community.

CPython bytecode changes
The bytecode now contains inline cache entries, which take the form of the newly-added CACHE instructions. Many opcodes expect to be followed by an exact number of caches, and instruct the interpreter to skip over them at runtime. Populated caches can look like arbitrary instructions, so great care should be taken when reading or modifying raw, adaptive bytecode containing quickened data.

New opcodes
ASYNC_GEN_WRAP , RETURN_GENERATOR and SEND , used in generators and co-routines.
COPY_FREE_VARS , which avoids needing special caller-side code for closures.
JUMP_BACKWARD_NO_INTERRUPT , for use in certain loops where handling interrupts is undesirable.
MAKE_CELL , to create Cell Objects .
CHECK_EG_MATCH and PREP_RERAISE_STAR , to handle the new exception groups and except* added in PEP 654 .
PUSH_EXC_INFO , for use in exception handlers.
RESUME , a no-op, for internal tracing, debugging and optimization checks.

Replaced opcodes
BINARY_* INPLACE_* -> BINARY_OP
CALL_FUNCTION CALL_FUNCTION_KW CALL_METHOD -> CALL KW_NAMES PRECALL PUSH_NULL
DUP_TOP DUP_TOP_TWO ROT_TWO ROT_THREE ROT_FOUR ROT_N -> COPY SWAP
JUMP_IF_NOT_EXC_MATCH -> CHECK_EXC_MATCH
JUMP_ABSOLUTE POP_JUMP_IF_FALSE POP_JUMP_IF_TRUE -> JUMP_BACKWARD POP_JUMP_BACKWARD_IF_* POP_JUMP_FORWARD_IF_*
SETUP_WITH SETUP_ASYNC_WITH -> BEFORE_WITH

Changed/removed opcodes
Changed MATCH_CLASS and MATCH_KEYS to no longer push an additional boolean value to indicate success/failure. Instead, None is pushed on failure in place of the tuple of extracted values.
Changed opcodes that work with exceptions to reflect them now being represented as one item on the stack instead of three (see gh-89874).
Removed COPY_DICT_WITHOUT_KEYS , GEN_START , POP_BLOCK , SETUP_FINALLY and YIELD_FROM .

Deprecated
This section lists Python APIs that have been deprecated in Python 3.11.
Deprecated C APIs are listed separately.

Language/Builtins
Chaining classmethod descriptors (introduced in bpo-19072) is now deprecated. It can no longer be used to wrap other descriptors such as property . The core design of this feature was flawed and caused a number of downstream problems. To “pass-through” a classmethod , consider using the wrapped attribute that was added in Python 3.10. (Contributed by Raymond Hettinger in gh-89519.)
Octal escapes in string and bytes literals with values larger than 0o377 (255 in decimal) now produce a DeprecationWarning . In a future Python version, they will raise a SyntaxWarning and eventually a SyntaxError . (Contributed by Serhiy Storchaka in gh-81548.)
The delegation of int() to trunc() is now deprecated. Calling int(a) when type(a) implements trunc() but not int() or index() now raises a DeprecationWarning . (Contributed by Zackery Spytz in bpo-44977.)

Modules
PEP 594 led to the deprecations of the following modules slated for removal in Python 3.13:
aifc, chunk, msilib, pipes, telnetlib
audioop, crypt, nis, sndhdr, uu
cgi, imghdr, nntplib, spwd, xdrlib
cgitb, mailcap, ossaudiodev, sunau
asynchat, asyncore and smtpd modules have been deprecated since at least Python 3.6. Their documentation and deprecation warnings have now been updated to note they will be removed in Python 3.12.
lib2to3 package and 2to3 tool are now deprecated and may not be able to parse Python 3.10 or newer.
Undocumented modules sre_compile , sre_constants and sre_parse are now deprecated.

Standard Library
The following have been deprecated in configparser since Python 3.2. Their deprecation warnings have now been updated to note they will be removed in Python 3.12:
the configparser.SafeConfigParser class
the configparser.ParsingError.filename property
the configparser.RawConfigParser.readfp() method
configparser.LegacyInterpolation has been deprecated in the docstring since Python 3.2 ...
[content truncated]

Original source

What’s New In Python 3.7

Editor:
Elvis Pranskevichus <

This article explains the new features in Python 3.7, compared to 3.6. Python 3.7 was released on June 27, 2018. For full details, see the
changelog.

Summary – Release Highlights

New syntax features:
•
PEP 563, postponed evaluation of type annotations.

Backwards incompatible syntax changes:
•
async and await are now reserved keywords.

New library modules:
•
contextvars: PEP 567 – Context Variables
•
dataclasses: PEP 557 – Data Classes
•
importlib.resources

New built-in features:
•
PEP 553, the new breakpoint() function.

Python data model improvements:
•
PEP 562, customization of access to module attributes.
•
PEP 560, core support for typing module and generic types.
•
the insertion-order preservation nature of dict objects has been declared to be an official part of the Python language spec.

Significant improvements in the standard library:
•
The asyncio module has received new features, significant usability and performance improvements.
•
The time module gained support for functions with nanosecond resolution.

CPython implementation improvements:
•
Avoiding the use of ASCII as a default text encoding:
◦
PEP 538, legacy C locale coercion
◦
PEP 540, forced UTF-8 runtime mode
•
PEP 552, deterministic .pycs
•
New Python Development Mode
•
PEP 565, improved DeprecationWarning handling

C API improvements:
•
PEP 539, new C API for thread-local storage

Documentation improvements:
•
PEP 545, Python documentation translations
•
New documentation translations: Japanese, French, and Korean.

This release features notable performance improvements in many areas. The
Optimizations section lists them in detail.

For a list of changes that may affect compatibility with previous Python releases please refer to the
Porting to Python 3.7 section.

New Features

PEP 563: Postponed Evaluation of Annotations

The advent of type hints in Python uncovered two glaring usability issues with the functionality of annotations added in PEP 3107 and refined further in PEP 526:
•
annotations could only use names which were already available in the current scope, in other words they didn’t support forward references of any kind; and
•
annotating source code had adverse effects on startup time of Python programs.

Both of these issues are fixed by postponing the evaluation of annotations. Instead of compiling code which executes expressions in annotations at their definition time, the compiler stores the annotation in a string form equivalent to the AST of the expression in question. If needed, annotations can be resolved at runtime using typing.get_type_hints(). In the common case where this is not required, the annotations are cheaper to store (since short strings are interned by the interpreter) and make startup time faster.

Usability-wise, annotations now support forward references, making the following syntax valid:

class C:

@classmethod

def from_string(cls, source: str) -> C:
    ...

def validate_b(self, obj: B) -> bool:
    ...

class B:
...

Since this change breaks compatibility, the new behavior needs to be enabled on a per-module basis in Python 3.7 using a future import:

from future import annotations

It will become the default in Python 3.10.

See also
PEP 563 – Postponed evaluation of annotations – PEP 563 – Postponed evaluation of annotations

PEP 538: Legacy C Locale Coercion

An ongoing challenge within the Python 3 series has been determining a sensible default strategy for handling the “7-bit ASCII” text encoding assumption currently implied by the use of the default C or POSIX locale on non-Windows platforms.

PEP 538 updates the default interpreter command line interface to automatically coerce that locale to an available UTF-8 based locale as described in the documentation of the new PYTHONCOERCECLOCALE environment variable. Automatically setting LC_CTYPE this way means that both the core interpreter and locale-aware C extensions (such as readline) will assume the use of UTF-8 as the default text encoding, rather than ASCII.

The platform support definition in PEP 11 has also been updated to limit full text handling support to suitably configured non-ASCII based locales.

As part of this change, the default error handler for stdin and stdout is now surrogateescape (rather than strict) when using any of the defined coercion target locales (currently C.UTF-8, C.utf8, and UTF-8). The default error handler for stderr continues to be backslashreplace, regardless of locale.

Locale coercion is silent by default, but to assist in debugging potentially locale related integration problems, explicit warnings (emitted directly on stderr) can be requested by setting PYTHONCOERCECLOCALE=warn. This setting will also cause the Python runtime to emit a warning if the legacy C locale remains active when the core interpreter is initialized.

While PEP 538’s locale coercion has the benefit of also affecting extension modules (such as GNU readline), as well as child processes (including those running non-Python applications and older versions of Python), it has the downside of requiring that a suitable target locale be present on the running system. To better handle the case where no suitable target locale is available (as occurs on RHEL/CentOS 7, for example), Python 3.7 also implements PEP 540: Forced UTF-8 Runtime Mode.

See also
PEP 538 – Coercing the legacy C locale to a UTF-8 based locale – PEP 538 – Coercing the legacy C locale to a UTF-8 based locale

PEP 540: Forced UTF-8 Runtime Mode

The new -X utf8 command line option and PYTHONUTF8 environment variable can be used to enable the Python UTF-8 Mode.

When in UTF-8 mode, CPython ignores the locale settings, and uses the UTF-8 encoding by default. The error handlers for sys.stdin and sys.stdout streams are set to surrogateescape.

The forced UTF-8 mode can be used to change the text handling behavior in an embedded Python interpreter without changing the locale settings of an embedding application.

While PEP 540’s UTF-8 mode has the benefit of working regardless of which locales are available on the running system, it has the downside of having no effect on extension modules (such as GNU readline), child processes running non-Python applications, and child processes running older versions of Python. To reduce the risk of corrupting text data when communicating with such components, Python 3.7 also implements PEP 540: Forced UTF-8 Runtime Mode.

The UTF-8 mode is enabled by default when the locale is C or POSIX, and the PEP 538 locale coercion feature fails to change it to a UTF-8 based alternative (whether that failure is due to PYTHONCOERCECLOCALE=0 being set, LC_ALL being set, or the lack of a suitable target locale).

See also
PEP 540 – Add a new UTF-8 mode – PEP 540 – Add a new UTF-8 mode

PEP 553: Built-in breakpoint()

PEP 553: Built-in breakpoint()

Python 3.7 includes the new built-in breakpoint() function as an easy and consistent way to enter the Python debugger.

Built-in breakpoint() calls sys.breakpointhook(). By default, the latter imports pdb and then calls pdb.set_trace(). By binding sys.breakpointhook() to the function of your choosing, breakpoint() can enter any debugger. Additionally, the environment variable PYTHONBREAKPOINT can be set to the callable of your debugger of choice. Set PYTHONBREAKPOINT=0 to completely disable built-in breakpoint().

See also
PEP 553 – Built-in breakpoint() – PEP 553 – Built-in breakpoint()

PEP 539: New C API for Thread-Local Storage

While Python provides a C API for thread-local storage support; the existing Thread Local Storage (TLS) API has used int to represent TLS keys across all platforms. This has not generally been a problem for officially support platforms, but that is neither POSIX-compliant, nor portable in any practical sense.

PEP 539 changes this by providing a new Thread Specific Storage (TSS) API to CPython which supersedes use of the existing TLS API within the CPython interpreter, while deprecating the existing API. The TSS API uses a new type Py_tss_t instead of int to represent TSS keys–an opaque type the definition of which may depend on the underlying TLS implementation. Therefore, this will allow to build CPython on platforms where the native TLS key is defined in a way that cannot be safely cast to int.

Note that on platforms where the native TLS key is defined in a way that cannot be safely cast to int, all functions of the existing TLS API will be no-op and immediately return failure. This indicates clearly that the old API is not supported on platforms where it cannot be used reliably, and that no effort will be made to add such support.

See also
PEP 539 – A New C-API for Thread-Local Storage in CPython – PEP 539 – A New C-API for Thread-Local Storage in CPython

PEP 562: Customization of Access to Module Attributes

Python 3.7 allows defining getattr() on modules and will call it whenever a module attribute is otherwise not found. Defining dir() on modules is now also allowed.

A typical example of where this may be useful is module attribute deprecation and lazy loading.

See also
PEP 562 – Module getattr and dir – PEP 562 – Module getattr and dir

PEP 564: New Time Functions With Nanosecond Resolution

The resolution of clocks in modern systems can exceed the limited precision of a floating-point number returned by the time.time() function and its variants. To avoid loss of precision, PEP 564 adds six new “nanosecond” variants of the existing timer functions to the time module:
•
time.clock_gettime_ns()
•
time.clock_settime_ns()
•
time.monotonic_ns()
•
time.perf_counter_ns()
•
time.process_time_ns()
•
time.time_ns()

The new functions return the number of nanoseconds as an integer value.

Measurements show that on Linux and Windows the resolution of time.time_ns() is approximately 3 times better than that of time.time().

See also
PEP 564 – Add new time functions with nanosecond resolution – PEP 564 – Add new time functions with nanosecond resolution

PEP 565: Show DeprecationWarning in main

PEP 565: Show DeprecationWarning in main

The default handling of DeprecationWarning has been changed such that these warnings are once more shown by default, but only when the code triggering them is running directly in the main module. As a result, developers of single file scripts and those using Python interactively should once again start seeing deprecation warnings for the APIs they use, but deprecation warnings triggered by imported application, library and framework modules will continue to be hidden by default.

As a result of this change, the standard library now allows developers to choose between three different deprecation warning behaviours:
•
FutureWarning: always displayed by default, recommended for warnings intended to be seen by application end users (e.g. for deprecated application configuration settings).
•
DeprecationWarning: displayed by default only in main and when running tests, recommended for warnings intended to be seen by other Python developers where a version upgrade may result in changed behaviour or an error.
•
PendingDeprecationWarning: displayed by default only when running tests, intended for cases where a future version upgrade will change the warning category to DeprecationWarning or FutureWarning.

Previously both DeprecationWarning and PendingDeprecationWarning were only visible when running tests, which meant that developers primarily writing single file scripts or using Python interactively could be surprised by breaking changes in the APIs they used.

See also
PEP 565 – Show DeprecationWarning in main – PEP 565 – Show DeprecationWarning in main

PEP 560: Core Support for typing module and Generic Types

PEP 560: Core Support for typing module and Generic Types

Initially PEP 484 was designed in such way that it would not introduce any changes to the core CPython interpreter. Now type hints and the typing module are extensively used by the community, so this restriction is removed. The PEP introduces two special methods class_getitem() and mro_entries(), these methods are now used by most classes and special constructs in typing. As a result, the speed of various operations with types increased up to 7 times, the generic types can be used without metaclass conflicts, and several long standing bugs in typing module are fixed.

See also
PEP 560 – Core support for typing module and generic types – PEP 560 – Core support for typing module and generic types

PEP 552: Hash-based .pyc Files

Python has traditionally checked the up-to-dateness of bytecode cache files (i.e., .pyc files) by comparing the source metadata (last-modified timestamp and size) with source metadata saved in the cache file header when it was generated. While effective, this invalidation method has its drawbacks. When filesystem timestamps are too coarse, Python can miss source updates, leading to user confusion. Additionally, having a timestamp in the cache file is problematic for build reproducibility and content-based build systems.

PEP 552 extends the pyc format to allow the hash of the source file to be used for invalidation instead of the source timestamp. Such .pyc files are called “hash-based”. By default, Python still uses timestamp-based invalidation and does not generate hash-based .pyc files at runtime. Hash-based .pyc files may be generated with py_compile or compileall.

Hash-based .pyc files come in two variants: checked and unchecked. Python validates checked hash-based .pyc files against the corresponding source files at runtime but doesn’t do so for unchecked hash-based pycs. Unchecked hash-based .pyc files are a useful performance optimization for environments where a system external to Python (e.g., the build system) is responsible for keeping .pyc files up-to-date.

See Cached bytecode invalidation for more information.

See also
PEP 552 – Deterministic pycs – PEP 552 – Deterministic pycs

PEP 545: Python Documentation Translations

PEP 545 describes the process of creating and maintaining Python documentation translations.

Three new translations have been added:
•
Japanese: https://docs.python.org/ja/
•
French: https://docs.python.org/fr/
•
Korean: https://docs.python.org/ko/

See also
PEP 545 – Python Documentation Translations – PEP 545 – Python Documentation Translations

Python Development Mode (-X dev)

The new -X dev command line option or the new PYTHONDEVMODE environment variable can be used to enable Python Development Mode. When in development mode, Python performs additional runtime checks that are too expensive to be enabled by default. See Python Development Mode documentation for the full description.

Other Language Changes

•
An await expression and comprehensions containing an async for clause were illegal in the expressions in formatted string literals due to a problem with the implementation. In Python 3.7 this restriction was lifted.
•
More than 255 arguments can now be passed to a function, and a function can now have more than 255 parameters. (Contributed by Serhiy Storchaka.)
•
bytes.fromhex() and bytearray.fromhex() now ignore all ASCII whitespace, not only spaces. (Contributed by Robert Xiao.)
•
str, bytes, and bytearray gained support for the new isascii() method, which can be used to test if a string or bytes contain only the ASCII characters. (Contributed by INADA Naoki.)
•
ImportError now displays module name and module file path when from ... import ... fails. (Contributed by Matthias Bussonnier.)
•
Circular imports involving absolute imports with binding a submodule to a name are now supported. (Contributed by Serhiy Storchaka.)
•
object.format(x, '') is now equivalent to str(x) rather than format(str(self), ''). (Contributed by Serhiy Storchaka.)
•
types.TracebackType can now be instantiated from Python code, and the tb_next attribute on tracebacks is now writable. (Contributed by Nathaniel J. Smith.)
•
When using the -m switch, sys.path[0] is now eagerly expanded to the full starting directory path, rather than being left as the empty directory. (Contributed by Nick Coghlan.)
•
The new -X importtime option or the PYTHONPROFILEIMPORTTIME environment variable can be used to show the timing of each module import. (Contributed by Inada Naoki.)

New Modules

contextvars

The new contextvars module and a set of new C APIs introduce support for context variables. Context variables are conceptually similar to thread-local variables. Unlike TLS, context variables support asynchronous code correctly.

The asyncio and decimal modules have been updated to use and support context variables out of the box. Particularly the active decimal context is now stored in a context variable, which allows decimal operations to work with the correct context in asynchronous code.

See also
PEP 567 – Context Variables – PEP 567 – Context Variables

dataclasses

The new dataclass() decorator provides a way to declare data classes. A data class describes its attributes using class variable annotations. Its constructor and other magic methods, such as repr, eq, and hash, are generated automatically.

Example:

@dataclass
class Point:
x: float
y: float
z: float = 0.0

p = Point(1.5, 2.5)
print(p)

produces "Point(x=1.5, y=2.5, z=0.0)"

See also
PEP 557 – Data Classes – PEP 557 – Data Classes

importlib.resources

The new importlib.resources module provides several new APIs and one new ABC for access to, opening, and reading resources inside packages. Resources are roughly similar to files inside packages, but they needn’t be actual files on the physical file system. Module loaders can provide a get_resource_reader() function which returns a importlib.abc.ResourceReader instance to support this new API. Built-in file path loaders and zip file loaders both support this.

Contributed by Barry Warsaw and Brett Cannon.

See also
importlib_resources – a PyPI backport for earlier Python versions.

Improved Modules

(argparse, asyncio, binascii, calendar, collections, compileall, concurrent.futures, contextlib, cProfile, crypt, datetime, dbm, decimal, dis, distutils, enum, functools, gc, hmac, http.client, http.server, idlelib and IDLE, importlib, io, ipaddress, itertools, locale, logging, math, mimetypes, msilib, multiprocessing, os, pathlib, pdb, py_compile, pydoc, queue, re, signal, socket, socketserver, sqlite3, ssl, string, subprocess, sys, time, tkinter, tracemalloc, types, unicodedata, unittest, unittest.mock, urllib.parse, uu, uuid, warnings, xml, xml.etree, xmlrpc.server, zipapp, zipfile)

C API Changes

A new API for thread-local storage has been implemented. See PEP 539 and Thread-specific storage API for a complete reference.

The new context variables functionality exposes a number of new C APIs.

The new PyImport_GetModule() function returns the previously imported module with the given name. (Contributed by Eric Snow.)

The new Py_RETURN_RICHCOMPARE macro eases writing rich comparison functions. (Contributed by Petr Victorin.)

The new Py_UNREACHABLE macro can be used to mark unreachable code paths. (Contributed by Barry Warsaw.)

The tracemalloc now exposes a C API through the new PyTraceMalloc_Track() and PyTraceMalloc_Untrack() functions. (Contributed by Victor Stinner.)

The new import__find__load__start and import__find__load__done static markers can be used to trace module imports. (Contributed by Christian Heimes.)

Various other C API changes and additions are described.

Build Changes

Support for building --without-threads has been removed. The threading module is now always available.

A full copy of libffi is no longer bundled for use when building the _ctypes module on non-OSX UNIX platforms. An installed copy of libffi is now required when building _ctypes on such platforms.

The Windows build process no longer depends on Subversion to pull in external sources, a Python script is used to download zipfiles from GitHub instead.

The ssl module requires OpenSSL 1.0.2 or 1.1 compatible libssl. OpenSSL 1.0.1 has reached end of lifetime and is no longer supported.

Optimizations

The overhead of calling many methods of various standard library classes implemented in C has been significantly reduced by porting more code to use the METH_FASTCALL convention.

Various optimizations have reduced Python startup time by 10% on Linux and up to 30% on macOS.

Method calls are now up to 20% faster due to the bytecode changes which avoid creating bound method instances.

asyncio module received a number of notable optimizations (get_event_loop reimplemented in C, Future callback management optimized, gather faster, sleep faster when delay is zero, etc.).

As a result of PEP 560 work, the import time of typing has been reduced by a factor of 7.

sorted() and list.sort() have been optimized for common cases to be up to 40-75% faster.

dict.copy() is now up to 5.5 times faster.

hasattr() and getattr() are now about 4 times faster when name is not found and obj does not override object.getattr or object.getattribute.

Many other optimizations described.

Other CPython Implementation Changes

Trace hooks may now opt out of receiving the line and opt into receiving the opcode events by setting f_trace_lines and f_trace_opcodes attributes on the frame being traced.

Fixed some consistency problems with namespace package module attributes.

The locals() dictionary now displays in the lexical order that variables were defined.

Various other implementation changes.

Deprecated Python Behavior

Yield expressions in comprehensions and generator expressions are now deprecated (will be SyntaxError in future).

Returning a subclass of complex from object.complex() is deprecated and will be an error in future Python versions.

Deprecated Python modules, functions and methods

(aifc, asyncio, collections, dbm, enum, gettext, importlib, locale, macpath, threading, socket, ssl, sunau, sys, wave) - many specific deprecations listed.

Deprecated functions and types of the C API

Various deprecated C API functions listed.

Platform Support Removals

FreeBSD 9 and older are no longer officially supported.

For full Unicode support, *nix platforms are now expected to provide at least one of C.UTF-8, C.utf8 or UTF-8 locales.

OpenSSL 0.9.8 and 1.0.1 are no longer supported.

API and Feature Removals

List of removed features and APIs (os.stat_float_times(), unknown escape handling in re.sub(), ntpath.splitunc(), verbose param of collections.namedtuple(), removal of keyword args for builtins like bool(), float(), list(), tuple(), etc.).

Module Removals

fpectl module has been removed.

Windows-only Changes

Improvements to the python launcher (py.exe) and related behavior.

Porting to Python 3.7

This section lists previously described changes and other bugfixes that may require changes to your code.

(Notable changes in Python 3.7.x maintenance releases and security fixes are listed, including changes in 3.7.1, 3.7.2, 3.7.6, 3.7.10, 3.7.11, and a notable security feature in 3.7.14.)

©

Last updated on Mar 17, 2026 (09:57 UTC).

Original source

What’s New In Python 3.8

Editor:
Raymond Hettinger

This article explains the new features in Python 3.8, compared to 3.7. Python 3.8 was released on October 14, 2019. For full details, see the changelog.

Summary – Release highlights

New Features

Assignment expressions

There is new syntax
:=
that assigns values to variables as part of a larger expression. It is affectionately known as “the walrus operator” due to its resemblance to the eyes and tusks of a walrus.

In this example, the assignment expression helps avoid calling
len()
twice:

if (n := len(a)) > 10:
print(f"List is too long ({n} elements, expected <= 10)")

A similar benefit arises during regular expression matching where match objects are needed twice, once to test whether a match occurred and another to extract a subgroup:

discount = 0.0
if (mo := re.search(r'(\d+)% d iscount', advertisement)):
discount = float(mo.group(1)) / 100.0

The operator is also useful with while-loops that compute a value to test loop termination and then need that same value again in the body of the loop:

Loop over fixed length blocks

while (block := f.read(256)) != '':
process(block)

Another motivating use case arises in list comprehensions where a value computed in a filtering condition is also needed in the expression body:

[clean_name.title() for name in names if (clean_name := normalize('NFC', name)) in allowed_names]

Try to limit use of the walrus operator to clean cases that reduce complexity and improve readability.

See PEP 572 for a full description.

(Contributed by Emily Morehouse in bpo-35224.)

Positional-only parameters

There is a new function parameter syntax
/
to indicate that some function parameters must be specified positionally and cannot be used as keyword arguments. This is the same notation shown by
help()
for C functions annotated with Larry Hastings’ Argument Clinic tool.

In the following example, parameters a and b are positional-only, while c or d can be positional or keyword, and e or f are required to be keywords:

def f(a, b, /, c, d, *, e, f):
print(a, b, c, d, e, f)

The following is a valid call:

f(10, 20, 30, d=40, e=50, f=60)

However, these are invalid calls:

f(10, b=20, c=30, d=40, e=50, f=60) # b cannot be a keyword argument
f(10, 20, 30, 40, 50, f=60) # e must be a keyword argument

One use case for this notation is that it allows pure Python functions to fully emulate behaviors of existing C coded functions. For example, the built-in
divmod()
function does not accept keyword arguments:

def divmod(a, b, /):
"Emulate the built in divmod() function"
return (a // b, a % b)

Another use case is to preclude keyword arguments when the parameter name is not helpful. For example, the builtin
len()
function has the signature
len(obj, /).
This precludes awkward calls such as:

len(obj='hello')

The "obj" keyword argument impairs readability

A further benefit of marking a parameter as positional-only is that it allows the parameter name to be changed in the future without risk of breaking client code. For example, in the
statistics
module, the parameter
dist
may be changed in the future. This was made possible with the following function specification:

def quantiles(dist, /, *, n=4, method='exclusive'):
...

Since the parameters to the left of
/
are not exposed as possible keywords, the parameters names remain available for use in
**kwargs
:

def f(a, b, /, **kwargs):
... print(a, b, kwargs)
...

f(10, 20, a=1, b=2, c=3)

a and b are used in two ways

10 20 {'a': 1, 'b': 2, 'c': 3}

This greatly simplifies the implementation of functions and methods that need to accept arbitrary keyword arguments. For example, here is an excerpt from code in the
collections
module:

class Counter(dict):
def init(self, iterable=None, /, **kwds):
# Note "iterable" is a possible keyword argument
...

See PEP 570 for a full description.

(Contributed by Pablo Galindo in bpo-36540.)

Parallel filesystem cache for compiled bytecode files

The new
PYTHONPYCACHEPREFIX
setting (also available as
-X pycache_prefix
) configures the implicit bytecode cache to use a separate parallel filesystem tree, rather than the default
pycache
subdirectories within each source directory.

The location of the cache is reported in
sys.pycache_prefix
(None indicates the default location in
pycache
subdirectories).

(Contributed by Carl Meyer in bpo-33499.)

Debug build uses the same ABI as release build

Python now uses the same ABI whether it’s built in release or debug mode. On Unix, when Python is built in debug mode, it is now possible to load C extensions built in release mode and C extensions built using the stable ABI.

Release builds and debug builds are now ABI compatible: defining the
Py_DEBUG
macro no longer implies the
Py_TRACE_REFS
macro, which introduces the only ABI incompatibility. The
Py_TRACE_REFS
macro, which adds the
sys.getobjects()
function and the
PYTHONDUMPREFS
environment variable, can be set using the new
./configure --with-trace-refs
build option. (Contributed by Victor Stinner in bpo-36465.)

On Unix, C extensions are no longer linked to libpython except on Android and Cygwin. It is now possible for a statically linked Python to load a C extension built using a shared library Python. (Contributed by Victor Stinner in bpo-21536.)

On Unix, when Python is built in debug mode, import now also looks for C extensions compiled in release mode and for C extensions compiled with the stable ABI. (Contributed by Victor Stinner in bpo-36722.)

To embed Python into an application, a new
--embed
option must be passed to
python3-config --libs --embed
to get
-lpython3.8
(link the application to libpython). To support both 3.8 and older, try
python3-config --libs --embed
first and fallback to
python3-config --libs
(without
--embed
) if the previous command fails.

Add a pkg-config
python-3.8-embed
module to embed Python into an application:

pkg-config python-3.8-embed --libs includes -lpython3.8. To support both 3.8 and older, try
pkg-config python-X.Y-embed --libs
first and fallback to
pkg-config python-X.Y --libs
(without
--embed
) if the previous command fails (replace
X.Y
with the Python version).

On the other hand,
pkg-config python3.8 --libs
no longer contains
-lpython3.8
. C extensions must not be linked to libpython (except on Android and Cygwin, whose cases are handled by the script); this change is backward incompatible on purpose. (Contributed by Victor Stinner in bpo-36721.)

f-strings support = for self-documenting expressions and debugging

Added an

specifier to f-strings. An f-string such as
f'{expr=}'
will expand to the text of the expression, an equal sign, then the representation of the evaluated expression. For example:

user = 'eric_idle'
member_since = date(1975, 7, 31)
f'{user=} {member_since=}'
"user='eric_idle' member_since=datetime.date(1975, 7, 31)"

The usual f-string format specifiers allow more control over how the result of the expression is displayed:

delta = date.today() - member_since
f'{user=!s} {delta.days=:,d}'
'user=eric_idle delta.days=16,075'

The = specifier will display the whole expression so that calculations can be shown:

print(f'{theta=} {cos(radians(theta)) =:.3f}')
theta=30 cos(radians(theta))=0.866

(Contributed by Eric V. Smith and Larry Hastings in bpo-36817.)

PEP 578: Python Runtime Audit Hooks

The PEP adds an Audit Hook and Verified Open Hook. Both are available from Python and native code, allowing applications and frameworks written in pure Python code to take advantage of extra notifications, while also allowing embedders or system administrators to deploy builds of Python where auditing is always enabled.

See PEP 578 for full details.

PEP 587: Python Initialization Configuration

The PEP 587 adds a new C API to configure the Python Initialization providing finer control on the whole configuration and better error reporting.

New structures:
• PyConfig
• PyPreConfig
• PyStatus
• PyWideStringList

New functions:
• PyConfig_Clear()
• PyConfig_InitIsolatedConfig()
• PyConfig_InitPythonConfig()
• PyConfig_Read()
• PyConfig_SetArgv()
• PyConfig_SetBytesArgv()
• PyConfig_SetBytesString()
• PyConfig_SetString()
• PyPreConfig_InitIsolatedConfig()
• PyPreConfig_InitPythonConfig()
• PyStatus_Error()
• PyStatus_Exception()
• PyStatus_Exit()
• PyStatus_IsError()
• PyStatus_IsExit()
• PyStatus_NoMemory()
• PyStatus_Ok()
• PyWideStringList_Append()
• PyWideStringList_Insert()
• Py_BytesMain()
• Py_ExitStatusException()
• Py_InitializeFromConfig()
• Py_PreInitialize()
• Py_PreInitializeFromArgs()
• Py_PreInitializeFromBytesArgs()
• Py_RunMain()

This PEP also adds _PyRuntimeState.preconfig (PyPreConfig type) and PyInterpreterState.config (PyConfig type) fields to these internal structures. PyInterpreterState.config becomes the new reference configuration, replacing global configuration variables and other private variables.

See Python Initialization Configuration for the documentation.

See PEP 587 for a full description.

(Contributed by Victor Stinner in bpo-36763.)

PEP 590: Vectorcall: a fast calling protocol for CPython

The Vectorcall Protocol is added to the Python/C API. It is meant to formalize existing optimizations which were already done for various classes. Any static type implementing a callable can use this protocol.

This is currently provisional. The aim is to make it fully public in Python 3.9.

See PEP 590 for a full description.

(Contributed by Jeroen Demeyer, Mark Shannon and Petr Viktorin in bpo-36974.)

Pickle protocol 5 with out-of-band data buffers

When
pickle
is used to transfer large data between Python processes in order to take advantage of multi-core or multi-machine processing, it is important to optimize the transfer by reducing memory copies, and possibly by applying custom techniques such as data-dependent compression.

The
pickle
protocol 5 introduces support for out-of-band buffers where PEP 3118-compatible data can be transmitted separately from the main pickle stream, at the discretion of the communication layer.

See PEP 574 for a full description.

(Contributed by Antoine Pitrou in bpo-36785.)

Other Language Changes

A continue statement was illegal in the finally clause due to a problem with the implementation. In Python 3.8 this restriction was lifted. (Contributed by Serhiy Storchaka in bpo-32489.)

The bool, int, and fractions.Fraction types now have an as_integer_ratio() method like that found in float and decimal.Decimal. This minor API extension makes it possible to write numerator, denominator = x.as_integer_ratio() and have it work across multiple numeric types. (Contributed by Lisa Roach in bpo-33073 and Raymond Hettinger in bpo-37819.)

Constructors of int, float and complex will now use the index() special method, if available and the corresponding method int(), float, or complex() is not available. (Contributed by Serhiy Storchaka in bpo-20092.)

Added support of \N{name} escapes in regular expressions:

notice = 'Copyright © 2019'
copyright_year_pattern = re.compile(r'\N{copyright sign}\s*(\d{4})')
int(copyright_year_pattern.search(notice).group(1))
2019

(Contributed by Jonathan Eunice and Serhiy Storchaka in bpo-30688.)

Dict and dictviews are now iterable in reversed insertion order using
reversed().
(Contributed by Rémi Lapeyre in bpo-33462.)

The syntax allowed for keyword names in function calls was further restricted. In particular, f((keyword)=arg) is no longer allowed. It was never intended to permit more than a bare name on the left-hand side of a keyword argument assignment term. (Contributed by Benjamin Peterson in bpo-34641.)

Generalized iterable unpacking in
yield
and
return
statements no longer requires enclosing parentheses. This brings the yield and return syntax into better agreement with normal assignment syntax.

(Contributed by David Cuthbert and Jordan Chapman in bpo-32117.)

When a comma is missed in code such as [(10, 20)(30, 40)], the compiler displays a SyntaxWarning with a helpful suggestion. This improves on just having a TypeError indicating that the first tuple was not callable. (Contributed by Serhiy Storchaka in bpo-15248.)

Arithmetic operations between subclasses of datetime.date or datetime.datetime and datetime.timedelta objects now return an instance of the subclass, rather than the base class. This also affects the return type of operations whose implementation (directly or indirectly) uses datetime.timedelta arithmetic, such as astimezone(). (Contributed by Paul Ganssle in bpo-32417.)

When the Python interpreter is interrupted by Ctrl-C (SIGINT) and the resulting KeyboardInterrupt exception is not caught, the Python process now exits via a SIGINT signal or with the correct exit code such that the calling process can detect that it died due to a Ctrl-C. Shells on POSIX and Windows use this to properly terminate scripts in interactive sessions. (Contributed by Google via Gregory P. Smith in bpo-1054041.)

Some advanced styles of programming require updating the types.CodeType object for an existing function. Since code objects are immutable, a new code object needs to be created, one that is modeled on the existing code object. With 19 parameters, this was somewhat tedious. Now, the new replace() method makes it possible to create a clone with a few altered parameters.

(Contributed by Victor Stinner in bpo-37032.)

For integers, the three-argument form of the pow() function now permits the exponent to be negative in the case where the base is relatively prime to the modulus. It then computes a modular inverse to the base when the exponent is -1, and a suitable power of that inverse for other negative exponents.

(Contributed by Mark Dickinson in bpo-36027.)

Dict comprehensions have been synced-up with dict literals so that the key is computed first and the value second. The guaranteed execution order is helpful with assignment expressions because variables assigned in the key expression will be available in the value expression. (Contributed by Jörn Heissler in bpo-35224.)

The object.reduce() method can now return a tuple from two to six elements long. Formerly, five was the limit. The new, optional sixth element is a callable with a (obj, state) signature. This allows the direct control over the state-updating behavior of a specific object. If not None, this callable will have priority over the object’s setstate() method. (Contributed by Pierre Glaser and Olivier Grisel in bpo-35900.)

New Modules

The new importlib.metadata module provides (provisional) support for reading metadata from third-party packages. For example, it can extract an installed package’s version number, list of entry points, and more.

(Contributed by Barry Warsaw and Jason R. Coombs in bpo-34632.)

Improved Modules

ast

AST nodes now have end_lineno and end_col_offset attributes, which give the precise location of the end of the node. New function ast.get_source_segment() returns the source code for a specific AST node. (Contributed by Ivan Levkivskyi in bpo-33416.)

asyncio

asyncio.run() has graduated from the provisional to stable API. This function can be used to execute a coroutine and return the result while automatically managing the event loop. Running python -m asyncio launches a natively async REPL. The exception asyncio.CancelledError now inherits from BaseException rather than Exception. On Windows, the default event loop is now ProactorEventLoop. ProactorEventLoop now also supports UDP and can be interrupted by KeyboardInterrupt. Added asyncio.Task.get_coro(), task naming, Happy Eyeballs support, and more. (Contributed by Yury Selivanov and others.)

builtins

The compile() built-in has been improved to accept the ast.PyCF_ALLOW_TOP_LEVEL_AWAIT flag. With this new flag passed, compile() will allow top-level await, async for and async with constructs that are usually considered invalid syntax. (Contributed by Matthias Bussonnier.)

collections

The _asdict() method for collections.namedtuple() now returns a dict instead of a collections.OrderedDict. (Contributed by Raymond Hettinger in bpo-35864.)

cProfile

The cProfile.Profile class can now be used as a context manager. (Contributed by Scott Sanderson in bpo-29235.)

csv

The csv.DictReader now returns instances of dict instead of a collections.OrderedDict. The tool is now faster and uses less memory while still preserving the field order. (Contributed by Michael Selik in bpo-34003.)

ctypes

On Windows, CDLL and subclasses now accept a winmode parameter to specify flags for the underlying LoadLibraryEx call. The default flags are set to only load DLL dependencies from trusted locations. (Contributed by Steve Dower in bpo-36085.)

datetime

Added new alternate constructors datetime.date.fromisocalendar() and datetime.datetime.fromisocalendar(). (Contributed by Paul Ganssle in bpo-36004.)

functools

functools.lru_cache() can now be used as a straight decorator. Added functools.cached_property() and functools.singledispatchmethod(). (Contributed by Raymond Hettinger and others.)

gc

get_objects() can now receive an optional generation parameter. (Contributed by Pablo Galindo in bpo-36016.)

gettext

Added pgettext() and its variants. (Contributed by Franz Glasner, Éric Araujo, and Cheryl Sabella in bpo-2504.)

gzip

Added the mtime parameter to gzip.compress() for reproducible output. A BadGzipFile exception is now raised instead of OSError for certain invalid gzip files. (Contributed by Guo Ci Teo; Filip Gruszczyński et al.)

IDLE and idlelib

Multiple UI and behavior improvements; see details. New in 3.8.1: toggle cursor blink, Escape key closes completion windows. (Contributed by various contributors.)

inspect

inspect.getdoc() can now find docstrings for slots if that attribute is a dict where the values are docstrings. (Contributed by Raymond Hettinger in bpo-36326.)

io

In development mode and in debug build, the io.IOBase finalizer now logs the exception if close() fails. (Contributed by Victor Stinner in bpo-18748.)

itertools

itertools.accumulate() added an initial keyword argument. (Contributed by Lisa Roach in bpo-34659.)

json.tool

Add option --json-lines to parse every input line as a separate JSON object. (Contributed by Weipeng Hong in bpo-31553.)

logging

Added a force keyword argument to logging.basicConfig(). (Suggested by Raymond Hettinger, implemented by Donghee Na.)

math

Added math.dist(), expanded math.hypot() to multiple dimensions, added math.prod(), math.perm(), math.comb(), math.isqrt(), and stricter argument types for math.factorial(). (Contributed by various contributors.)

mmap

mmap.mmap class now has a madvise() method. (Contributed by Zackery Spytz in bpo-32941.)

multiprocessing

Added new multiprocessing.shared_memory module. On macOS, spawn start method is now default. (Contributed by Davin Potts and Victor Stinner.)

os

Added add_dll_directory() on Windows, os.memfd_create(), improved reparse point handling on Windows, os.readlink() can read directory junctions. (Contributed by Steve Dower and others.)

os.path / pathlib

Functions that return boolean results now return False instead of raising ValueError or Unicode errors for unrepresentable paths. expanduser() on Windows prefers USERPROFILE. pathlib.Path.link_to() added. (Contributed by Serhiy Storchaka et al.)

pickle

pickle extensions subclassing the C-optimized Pickler can now override reducer_override(). (Contributed by Pierre Glaser and Olivier Grisel.)

plistlib

Added plistlib.UID and NSKeyedArchiver support. (Contributed by Jon Janzen.)

pprint

Added sort_dicts parameter and pprint.pp() convenience function. (Contributed by Rémi Lapeyre.)

py_compile

py_compile.compile() now supports silent mode. (Contributed by Joannah Nanjekye.)

shlex

Added shlex.join(). (Contributed by Bo Bayles.)

shutil

shutil.copytree() now accepts dirs_exist_ok; shutil.make_archive() defaults to pax; rmtree on Windows removes directory junctions without recursing. (Contributed by various.)

socket

Added create_server() and has_dualstack_ipv6(), implemented if_* functions on Windows. (Contributed by Giampaolo Rodolà and Zackery Spytz.)

ssl

Added post_handshake_auth and verify_client_post_handshake() for TLS 1.3. (Contributed by Christian Heimes.)

statistics

Added statistics.fmean(), geometric_mean(), multimode(), quantiles(), NormalDist, and examples. (Contributed by Raymond Hettinger and others.)

sys

Add new sys.unraisablehook() function to control handling of "unraisable exceptions". (Contributed by Victor Stinner.)

tarfile

tarfile module now defaults to pax format for new archives. (Contributed by C.A.M. Gerlach.)

threading

Add threading.excepthook(), threading.get_native_id(), and native_id attribute. (Contributed by Victor Stinner and Jake Tesler.)

tokenize

tokenize now implicitly emits a NEWLINE token when input lacks a trailing newline. (Contributed by Ammar Askar.)

tkinter

Added selection_* and moveto(), transparency_get/set, and other improvements. (Contributed by various.)

time

Added CLOCK_UPTIME_RAW for macOS 10.12. (Contributed by Joannah Nanjekye.)

typing

Multiple typing enhancements: TypedDict (PEP 589), Literal types (PEP 586), Final (PEP 591), Protocols (PEP 544), typing.SupportsIndex, typing.get_origin(), typing.get_args(), and more.

unicodedata

Upgraded to Unicode 12.1.0 and added is_normalized(). (Contributed by Max Belanger et al.)

unittest

Added AsyncMock, addModuleCleanup, addClassCleanup, support for coroutines as test cases (IsolatedAsyncioTestCase), and examples. (Contributed by Lisa Roach and others.)

venv

venv now includes an Activate.ps1 script on all platforms for PowerShell Core. (Contributed by Brett Cannon.)

weakref

weakref.proxy() now supports matrix multiplication operators @ and @=. (Contributed by Mark Dickinson.)

xml / xmlrpc

Security mitigations: xml.dom.minidom and xml.sax no longer process external entities by default; ElementTree improvements and canonicalize() added; xmlrpc.client.ServerProxy supports headers argument.

Optimizations

Multiple performance improvements across subprocess, shutil, pickle, UUID memory footprint, operator.itemgetter(), collections.namedtuple lookups, list constructor allocation, class variable writes, builtin call overhead reductions, and LOAD_GLOBAL opcode caching (~40% faster).

Build and C API Changes

Numerous changes: sys.abiflags default empty, header reorganization, macros converted to inline functions, PyByteArray_Init/Fini removed, integer conversion now prefers index(), PyObject_Init() reference changes for heap-allocated types, new PyCode_NewWithPosOnlyArgs(), Py_SetPath() sets sys.executable to full path, and many more.

Deprecated

Multiple deprecations including distutils bdist_wininst, ElementTree getchildren/getiterator, passing non-instance to loop.set_default_executor(), deprecated getitem implementations, typing.NamedTuple._field_types, ast Num/Str/Bytes/NameConstant/Ellipsis classes, @asyncio.coroutine decorator, explicit loop parameter in many asyncio APIs, and others.

API and Feature Removals

Several removals: macpath, platform.popen(), time.clock(), pyvenv script, cgi functions moved, tarfile.filemode removed, XMLParser html argument removed, unicode_internal codec removed, sqlite3 Cache/Statement objects not exposed, fileinput.bufsize removed, sys.set_coroutine_wrapper/get_coroutine_wrapper removed, and others.

Porting to Python 3.8

Guidance on changes in Python behavior, Python API, C API, CPython bytecode changes, demos and tools, and notable changes in maintenance releases 3.8.1 through 3.8.17 (security fixes, macOS Big Sur and Apple Silicon support backports, urllib.parse safety fixes, tarfile extraction filters, integer conversion limits for CVE mitigation, etc.).

Last updated on Mar 17, 2026 (09:57 UTC).

Original source