《dive into python3》 笔记摘录

0、In Python 2, the / operator usually meant integer division, but you could make it behave like floating point division by including a special directive in your code. In Python 3, the / operator always means floating point division.

1、list can hold arbitrary objects and can expand dynamically as new items are added. A list is an ordered set of items.

2、A tuple is an  immutable list. A tuple can not be changed in any way once it is created.

3、A set is an  unordered “bag”  of unique values. A single set can contain values of any immutable datatype.

4、A dictionary is an  unordered set of key-value pairs. keys are  unique and  immutable

5、import os, glob, humansize

     

     metadata_list = [(f, os.stat(f)) for f in glob.glob('*test*.py')]

     metadata_dict = {f:os.stat(f) for f in glob.glob('*')}

     humansize_dict = {os.path.splitext(f)[0]:humansize.approximate_size(meta.st_size) \
                              for f, meta in metadata_dict.items() if meta.st_size > 6000}

     

     a_set = {2**x for x in range(10)}

6、Bytes are not characters; bytes are bytes. Characters are an abstraction. A string is a sequence of those 

abstractions. In Python 3, all strings are  immutable sequences of Unicode characters.The built-in len() 

function returns the length of the string, i.e. the  number of characters. A string is like a tuple of characters.

An  immutable sequence of numbers-between-0-and-255 is called a bytes object.

Each item in a string is a string, each item in a byte array is an integer.

aBuf = b'\xEF\xBB\xBF'

aBuf[-1] #191

aBuf[-1:] #b'\xbf'  byte array

7、To define a bytes object, use the  b' ' “byte literal” syntax. Each byte within the byte literal can be an 

ASCII character or an encoded hexadecimal number from \x00 to \xff (0–255).To convert a bytes object into 

a mutable bytearray object, use the built-in bytearray() function.

8、bytes objects have a  decode() method that takes acharacter encoding and returns a string, and strings 

have an  encode() method that takes a characterencoding and returns a bytes object. 

9、'1MB = 1000{0.modules[humansize].SUFFIXES[1000][0]}'.format(sys)  #compound field names

10、“The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string.”

11、Within a replacement field, a colon (:) marks the start of the format specifier.

12、compact regular expressions

import re

s = '100 BROAD ROAD APT. 3'
 re.sub(r'\bROAD\b', 'RD.', s)

#  re.sub(b'\bROAD\b', 'RD.', s)

# search bytes

pattern = '^M{0,3}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$'
re.search(pattern, 'MDLV')

phonePattern = re.compile(r'(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$')

#Putting it all in parentheses means “match exactly three numeric digits, and then remember them as a #group that I can ask for later”. 
phonePattern.search('(800)5551212 ext. 1234').groups() 
('800', '555', '1212', '1234')
>>> phonePattern.search('800-555-1212').groups() 
('800', '555', '1212', '')
>>> phonePattern.search('work 1-(800) 555.1212 #1234')

('800', '555', '1212', '1234')

if phonePattern.match('800-555-1212'):

     do_something

if phonePattern.search('800-555-1212'):

     do_something

re.sub('([^aeiou])y$', r'\1ies', 'vacancy') 

'vacancies'

# \1, which means “hey, that first group you remembered? put it right here.” 

re.findall('[0-9]+', '16 2-by-4s in rows of 8') 
['16', '2', '4', '8']
re.findall('[A-Z]+', 'SEND + MORE == MONEY') 
['SEND', 'MORE', 'MONEY']

re.findall(' s.*? s', "The sixth sick sheikh's sixth sheep's sick.") 

# (.*?) means the shortest possible series of any character

[' sixth s', " sheikh's s", " sheep's s"]  #doesn’t return overlapping matches. 

13、verbose regular expressions

Python allows you to do this with something called verbose regular expressions. A verbose regular expression
is different from a compact regular expression in two ways:

• Whitespace is ignored. Spaces, tabs, and carriage returns are not matched as spaces, tabs, and carriage
returns. They’re not matched at all. (If you want to match a space in a verbose regular expression, you’ll
need to escape it by putting a backslash in front of it.)

• Comments are ignored. A comment in a verbose regular expression is just like a comment in Python code:
it starts with a # character and goes until the end of the line. In this case it’s a comment within a multi-line
string instead of within your source code, but it works the same way.

pattern = '''
^                                    # beginning of string
M{0,3}                              # thousands - 0 to 3 Ms
(CM|CD|D?C{0,3})           # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 Cs),
                                   # or 500-800 (D, followed by 0 to 3 Cs)
(XC|XL|L?X{0,3})           # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 Xs),
                                   # or 50-80 (L, followed by 0 to 3 Xs)
(IX|IV|V?I{0,3})                # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 Is),
                                   # or 5-8 (V, followed by 0 to 3 Is)
$                                    # end of string
'''
 re.search(pattern, 'M', re.VERBOSE) 

phonePattern = re.compile(r'''
                                   # don't match beginning of string, number can start anywhere
(\d{3})                          # area code is 3 digits (e.g. '800')
\D*                               # optional separator is any number of non-digits
(\d{3})                          # trunk is 3 digits (e.g. '555')
\D*                               # optional separator
(\d{4})                          # rest of number is 4 digits (e.g. '1212')
\D*                               # optional separator
(\d*)                          # extension is optional and can be any number of digits
$                               # end of string
''', re.VERBOSE)

phonePattern.search('work 1-(800) 555.1212 #1234').groups()

14、regular expressions

• ^ matches the beginning of a string.
• $ matches the end of a string.
• \b matches a word boundary.
• \d matches any numeric digit.
• \D matches any non-numeric character.
• x? matches an optional x character (in other words, it matches an x zero or one times).
• x* matches x zero or more times.
• x+ matches x one or more times.
• x{n,m} matches an x character at least n times, but not more than m times.
• (a|b|c) matches exactly one of a, b or c.
• (x) in general is a  remembered group. You can get the value of what matched by using the groups() methodof the object returned by re.search.

15、This technique of using the values of outside parameters within a dynamic function is called  closures. 

16、The  with statement creates what’s called a context: when the with block ends, Python will automatically close the file, even if an exception is raised inside the with block.

There’s nothing file-specific about the with statement; it’s just a generic framework for creating runtime
contexts and telling objects that they’re entering and exiting a runtime context. If the object in question is a
stream object, then it does useful file-like things (like closing the file automatically). But that behavior is
defined in the stream object, not in the with statement. 

17、The first argument to the  split() method is  None, which means “split on any whitespace (tabs or spaces, it makes no difference).” The second argument is 3, which means “split on whitespace 3 times, then leave the rest of the line alone.” 

18、The presence of the  yield x keyword in function body means that this is not a normal function. It is a special kind of function which generates values one at a time. You can think of it as a  resumable function. Calling it will return a  generator that can be used to generate successive values of x. The  next() function takes a generator object and returns its next value.

def fib(max):
    n, a, b = 0, 0, 1
    while n < max:
        yield b
        a, b = b, a + b
        n = n + 1

"yield" pause a function, and "next()" resumes where it left off.

Generators are just a simple form of iterators. A function that yields values is a nice, compact way of building an  iterator without building an iterator. File objects are iterators too! It’s iterators all the way down.

This is a useful idiom: pass a generator to the list() function, and it will iterate through the entire
generator (just like the  for loop) and return a list of all the values.

The for loop will automatically call the next() function to get values from the  generator and assign them to the for loop index variable.

实际上还有一种创建generator 的简单方法 ：

g = (x * x for x in range(10)) 注意与列表生成式 g = [ ... ] 区分

g.__next__() /*in Python 2, it's g.next()*/ or next(g) or for n in g : print(n)

19、‘ pass' is a Python reserved word that just means “move along, nothing to see here”. 

20、The first argument of every class method, including the  __init__() method, is always a reference to the

current instance of the class. By convention, this argument is named  self . 

21、An  iterator is just a class that defines an  __iter__() method. 

 Python Code 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

class Fib:      '''iterator that yields numbers in the Fibonacci sequence'''      def  __init__(self, maxn):         self.maxn = maxn      def  __iter__(self):         self.a =  0         self.b =  1          return self      def __next__(self):         fib = self.a          if fib > self.maxn:              raise  StopIteration         self.a, self.b = self.b, self.a + self.b          return fib for n  in Fib( 1000):      print(n, end= ' ')

After performing beginning-of-iteration initialization, the __iter__() method can return any object that implements a  __next__() method. The __next__() method is called whenever someone calls  next() on an iterator of an instance of a class.

iter(object) calls object.__iter__(), return an iterator object; 

next(iterator_object) calls iterator_object.__next__(), return a value;

 C++ Code 

1 2 3

for n in Fib( 1000):      print(n, end= ' ')

a for loop calls __init__() (if the object exists, ignore) and __iter__() once, but calls __next__() several times until encounter raise  StopIteration exception. 

When the __next__() method raises a StopIteration exception, this signals to the caller that the iteration
is exhausted. Unlike most exceptions, this is not an error; it’s a normal condition that just means that the
iterator has no more values to generate. If the caller is a for loop, it will notice this StopIteration
exception and gracefully  exit the loop. 

22、when the variable was  not defined within any method. It’s defined at the  class level. It’s a  class variable, and although you can access it just like an instance variable (self.rules_filename), it is shared across all instances of the same class.

23、A  generator expression is like an anonymous function that yields values. The expression itself looks like a list comprehension, but it’s wrapped in  parentheses instead of square brackets.

unique_characters = {'E', 'D', 'M', 'O', 'N', 'S', 'R', 'Y'}
 gen = (ord(c) for c in unique_characters)

If you like, you can iterate through all the possible values and return a tuple, list, or set, by passing the
generator expression to tuple(), list(), or set(). In these cases, you don’t need an extra set of
parentheses — just pass the “bare” expression ord(c) for c in unique_characters to the tuple()
function, and Python figures out that it’s a generator expression.

tuple(ord(c) for c in unique_characters) 
(69, 68, 77, 79, 78, 83, 82, 89)

24、The  itertools.permutations() function doesn’t have to take a list. It can take any sequence — even a string.The permutations() function takes a sequence and a number, which is the number of items you want in each smaller group. The function returns an iterator.

The  itertools.combinations() function returns an iterator containing all the possible combinations of the
given sequence of the given length.

 

The itertools. groupby() function takes a sequence and a key function, and returns an iterator that
generates pairs. Each pair contains the result of key_function(each item) and another iterator containing
all the items that shared that key result.

The itertools.groupby() function only works if the input sequence is already sorted by the grouping function. 

The  itertools.chain() function takes two iterators and returns an iterator that contains all the items
from the first iterator, followed by all the items from the second iterator. (Actually, it can take any number
of iterators, and it chains them all in the order they were passed to the function.)

25、 rstrip() string method to strip trailing whitespace from each line. (Strings also have an  lstrip() method to strip leading whitespace, and a  strip() method which strips both.)

26、zip()、tuple()、dict()、translate()、eval()

characters = ('S', 'M', 'E', 'D', 'O', 'N', 'R', 'Y') guess = ('1', '2', '0', '3', '4', '5', '6', '7')

 tuple(zip(characters, guess)) 
(('S', '1'), ('M', '2'), ('E', '0'), ('D', '3'), ('O', '4'), ('N', '5'), ('R', '6'), ('Y', '7'))
 dict(zip(characters, guess)) 
{'E': '0', 'D': '3', 'M': '2', 'O': '4', 'N': '5', 'S': '1', 'R': '6', 'Y': '7'}

'SEND + MORE == MONEY'.translate(translation_table) 
'1053 + 2460 == 24507'

The second and third parameters passed to the  eval() function act as the global and local namespaces for
evaluating the expression. 

The  subprocess module allows you to run arbitrary shell commands and get the result as a Python string.

eval("__import__('subprocess').getoutput('rm -rf /')",  {"__builtins__":None}, {}) 

#error. the __import__() function is also a builtin function

27、Running the script runs  unittest.main(), which runs each test case. Each test case is a method within a

class in xxxtest.py. There is no required organization of these test classes; they can each contain a single
test method, or you can have one class that contains multiple test methods. The only requirement is that
each test class must inherit from  unittest.TestCase.

The unittest.TestCase class provides the  assertRaises method, which takes the following arguments: the
exception you’re expecting, the function you’re testing, and the arguments you’re passing to that function. (If the function you’re testing takes more than one argument, pass them all to assertRaises, in order, and it
will pass them right along to the function you’re testing.)

28、It is important to understand that modules are only imported once, then cached. If you import an already-imported module, it does nothing. 

29、a file on disk is a sequence of bytes.The default encoding is platform dependent.

30、Python has a built-in function called  open(). The open() function returns a  stream object, which has methods and attributes for getting information about and manipulating a stream of characters.

31、Once you open a file (with the correct encoding), reading from it is just a matter of calling the stream
object’s  read() method. The result is a string.The read() method can take an optional parameter, the number of characters to read.

32、The  seek() and  tell() methods always count  bytes, but since you opened this file as text, the read() method counts  characters. Chinese characters require multiple bytes to encode in UTF -8 .

33、The stream object file still exists; calling its  close() method doesn’t destroy the object itself. But it’s

not terribly useful.Closed stream objects do have one useful attribute: the  closed  attribute will confirm that the file is closed.

34、Read a file one line at a time

 Python Code 

1 2 3 4 5

line_number =  0 with  open( 'examples/favorite-people.txt', encoding= 'utf-8')  as a_file:       for a_line  in a_file:          line_number +=  1          print( '{:>4} {}'. format(line_number, a_line. rstrip()))

the stream object is also an  iterator which spits out a single line every time you ask for a value.

The format specifier {:>4} means “print this argument right-justified within 4 spaces.” 

The  rstrip() string method removes the trailing whitespace, including the carriage return characters.

35、Reading a “string” from a  text file only works because you told Python what encoding to use to read a stream of bytes and convert it to a string. 

36、Opening a file in  binary mode is simple but subtle. The only difference from opening it in text mode is that the mode parameter contains a  'b' character. a binary stream object has no  encoding attribute.

Reading a file in “binary” mode? You’ll get a stream of bytes. Fetching a web page? Calling a web API ? They return a stream of bytes, too.

37、Since you opened the file in binary mode, the read() method takes the number of  bytes to read, not the number of characters.

38、As long as your functions take a stream object and simply call the object’s read() method, you can handle any  input source that acts like a file, without specific code to handle each kind of input.

39、 io.StringIO lets you treat a string as a text file. There’s also a  io.BytesIO class, which lets you treat a byte array as a binary file.

40、The  gzip module lets you create a stream object for reading or writing a gzip-compressed file.

You should always open gzipped files in binary mode.

 

import gzip
with gzip.open('out.log.gz', mode='wb') as z_file: 
     z_file.write('A nine mile walk is no joke, especially in the rain.'.encode('utf-8'))

41、The print function adds a carriage return to the end of the string you’re printing, and calls  sys.stdout.write.

42、Any class can be a  context manager by defining two special methods:  __enter__() and  __exit__().

43、The ElementTree library is part of the Python standard library, in  xml.etree.ElementTree

ElementTree represents XML elements as  {namespace}localname.

In the ElementTree API, an element acts like a  list. The items of the list are the element’s children.

XML isn’t just a collection of elements; each element can also have its own set of  attributes（.attrib）. Once you have a reference to a specific element, you can easily get its attributes as a Python dictionary.

In a boolean context, ElementTree element objects will evaluate to False if they contain no children (i.e. if  len(element) is 0). This means that if element.find('...') is not testing whether the find() method found a matching element; it’s testing whether that matching element has any child elements! To test whether the find() method returned an element, use  if element.find('...') is not None

44、The time module contains a data structure  (time_struct) to represent a point in time (accurate to one

millisecond) and functions to manipulate time structs. The  strptime()  function takes a formatted string an
converts it to a time_struct. 

45、The  dump() function in the  pickle module takes a serializable Python data structure, serializes it into a

binary, Python-specific format using the latest version of the pickle protocol, and saves it to an open file.

The  pickle.load() function takes a stream object, reads the serialized data from the stream, creates a new
Python object, recreates the serialized data in the new Python object, and returns the new Python object.

The  pickle.dumps() function (note the 's' at the end of the function name) performs the same
serialization as the pickle.dump() function. Instead of taking a stream object and writing the serialized data
to a file on disk, it simply returns the serialized data.

The  pickle.loads() function (again, note the 's' at the end of the function name) performs the same
deserialization as the pickle.load() function. Instead of taking a stream object and reading the serialized
data from a file, it takes a  bytes object containing serialized data, such as the one returned by the
pickle.dumps() function.

46、Like the pickle module, the  json module defines a dump() function which takes a Python data structure

and a writeable stream object. The dump() function serializes the Python data structure and writes it to the
stream object. Doing this inside a with statement will ensure that the file is closed properly when we’re
done.

JSON is a  text-based format. Always open JSON files in text mode with a UTF -8 character encoding.

JSON doesn’t distinguish between  tuples and lists; it only has a single list-like datatype, the array, and the json module silently converts both tuples and lists into JSON arrays during serialization. For most uses, you can ignore the difference between tuples and lists, but it’s something to keep in mind as you work with the json module.

47、The  time.asctime() function will convert that nasty-looking time.struct_time into the string 'Fri Mar 27 22:20:42 2009'.

We can use the list() function to convert the bytes object into a list of integers. So b'\xDE\xD5\xB4\xF8' becomes [222, 213, 180, 248]. 

import customserializer 
 with open('entry.json', 'w', encoding='utf-8') as f: 
      json.dump(entry, f, default=customserializer.to_json)   #shell 1

with open('entry.json', 'r', encoding='utf-8') as f:
     entry = json.load(f, object_hook=customserializer.from_json)       #shell 2

48、The  urllib.request.urlopen().read() method always returns a bytes object, not a string. Remember, bytes are  bytes;characters are an abstraction. HTTP servers don’t deal in abstractions. If you request a resource, you get bytes. If you want it as a string, you’ll need to determine the character encoding and explicitly convert it to a string.

The  response returned from the  urllib.request.urlopen() function contains all the  HTTP headers the
server sent back. download the actual data by calling  response.read()

49、The primary interface to  httplib2 is the  Http object.Once you have an Http object, retrieving data is as simple as calling the  request() method with the address of the data you want. This will issue an  HTTP GET request for that URL . 

The request() method returns two values. The first is an  httplib2.Response object, which contains all
the  HTTP headers the server returned. For example, a status code of 200 indicates that the request was
successful.

 The content variable contains the  actual data that was returned by the HTTP server. The data is returned
as a bytes object, not a string. If you want it as a string, you’ll need to determine the character encoding
and convert it yourself.

you should always create an  httplib2.Http object with a directory name. Caching is the reason.

httplib2 allows you to add arbitrary HTTP headers to any outgoing request. In order to bypass all caches
(not just your local disk cache, but also any caching proxies between you and the remote server), add a 

no-cache header in the headers dictionary.

HTTP defines  Last-Modified and  Etag headers for this purpose. These headers are called  validators. If the local cache is no longer fresh(expired), a client can send the validators with the next request to see if the data has actually changed. If the data hasn’t changed, the server sends back a 304 status code and no data. 

which caused httplib2 to look in its cache.

httplib2 sends the ETag validator back to the server in the  If-None-Match header.
 httplib2 also sends the Last-Modified validator back to the server in the  If-Modified-Since header.

50、Python comes with a utility function to URL -encode a dictionary: urllib.parse. urlencode()

Store your username and password with the  add_credentials() method. 

51、If Python sees an  __init__.py file in a directory, it assumes that all of the files in that directory are part of the same  module. The module’s name is the name of the directory. Files within the directory can reference other files within the same directory, or even within subdirectories.  But the entire collection of files is presented to other Python code as a single module — as if all the functions and classes were in a single .py file.

The __init__.py file doesn’t need to define anything; it can literally be an empty file. Or you can use it to define your main entry point functions. Or you put all your functions in it. Or all but one.

A directory with an __init__.py file is always treated as a  multi-file module.
Without an __init__.py file, a directory is just a directory of unrelated .py files.

52、Within lists, tuples, sets, and dictionaries, whitespace can appear before and after commas with no ill effects.

53、In Python 2, the global  file() function was an alias for the open() function, which was the standard way of opening text files for reading. In Python 3, the global file() function no longer exists, but the open() function still exists.

54、In Python 2, a string was an array of bytes whose character encoding was tracked separately. If you wanted Python 2 to keep track of the character encoding, you had to use a Unicode string  (u'') instead. But in Python 3, a string is always what Python 2 called a Unicode string — that is, an array of Unicode characters
(of possibly varying byte lengths). 

55. 类似c++ operator() 即函数对象的功能：假设login 现在是个Form对象，login() 会调用__call__()函数

即深拷贝了一份自身并返回，故可以看见 login_form=login() 的用法。

56. yield 与协程的关系

传统的生产者-消费者模型是一个线程写消息，一个线程取消息，通过锁机制控制队列和等待，但一不小心就可能死锁。如果改用协程，生产者生产消息后，直接通过yield 跳转到消费者开始执行，待消费者执行完毕后，切换回生产者继续生产，效率极高：

generator. send ( value )

Resumes the execution and “sends” a value into the generator function. The value argument becomes the result of the current yield expression. The send() method returns the next value yielded by the generator, or raises StopIteration if the generator exits without yielding another value. When send() is called to start the generator, it must be called with None as the argument, because there is no yield expression that could receive the value.

 Python Code 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

import time def consumer():     r =  ''      while  True:         n =  yield r          if  not n:              return          print( '[CONSUMER] Consuming %s...' % n)         time.sleep( 1)         r =  '200 OK' def produce(c):     c. next()    # 使用 send(None) 或 next() 启动协程     n =  0      while n <  5:         n = n +  1          print( '[PRODUCER] Producing %s...' % n)         r = c.send(n)  # 向协程发送消息,使其恢复执行          print( '[PRODUCER] Consumer return: %s' % r)     c. close()  # 关闭协程,使其退出。或 c.throw() 使其引发异常 if __name__== '__main__':     c = consumer()  # 函数返回协程对象     produce(c)

57. yield 与 contextmanage 的关系

 Python Code 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

@contextlib.contextmanager def some_generator():           try:          yield       finally:          with some_generator()  as :      也就是:           try:          =                finally:

 Python Code 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

>>>  import pymongo >>>  class Operation( object): ...      def  __init__(self, database, ...                  host= 'localhost', port= 27017): ...         self._db = pymongo.MongoClient( ...                       host, port)[database] ...      def __enter__(self): ...          return self._db ...      def __exit__(self, exc_type, exc_val, exc_tb): ...         self._db.connection.disconnect() ... >>>  with Operation(database= 'test')  as db: ...      print db.test.find_one() ... >>> @contextmanager ...  def operation(database, host= 'localhost',                    port= 27017): ...     db = pymongo.MongoClient(host, port)[database] ...      yield db ...     db.connection.disconnect() ...  >>>  import pymongo >>>  with operation( 'test')  as db: ...      print(db.test.find_one()) ...

摘自《dive into python3》