python 常用

Python setup.py develop vs install setuptools - Python setup.py develop vs install - Stack Overflow

python setup.py debug
python3 setup.py bdist_wheel
python3 -m pip install dist/*.whl

export TF_FORCE_GPU_ALLOW_GROWTH=true
f-string包含单个花括号字符：

>>> print(f"{{")
{
>>> print(f"}}")
}
>>> print(f"{{}}")
{}
>>> print(f"{{foo}}")
{foo}

from importlib import import_module
import_module
getattr

unittest

import unittest

class WidgetTestCase(unittest.TestCase):
    def setUp(self):
        pass

    def tearDown(self):
        pass

    def test1(self):
        print("test1")
    def test2(self):
        print("test2")

python -m unittest discover -s ${SCRIPT_DIR}/tests/python/ -p "*_test.py"
https://docs.python.org/3/library/unittest.html

pip install tensorflow==1.14 -i https://pypi.tuna.tsinghua.edu.cn/simple
source /mnt/xx/anaconda3/bin/activate 1.15
 

sort dict by key or value

d = {2:3, 1:89, 4:5, 3:0}
dict(sorted(d.items()))

dict(sorted(x.items(), key=lambda item: item[1]))

python - How can I sort a dictionary by key? - Stack Overflow

python - How do I sort a dictionary by value? - Stack Overflow

执行shell命名并获取结果

import subprocess

p = subprocess.Popen("nvidia-smi -L", shell=True, stdout=subprocess.PIPE).communicate()[0]
print(p.decode().split("\n"))

Popen shell=False方法
subprocess.Popen(["ls", "-l"], shell=False, ...)

import os
import sys
import subprocess
import glob

    cmd_str_list = []
    cmd_str_list.append("converter_lite")
    cmd_str_list.append(f"--fmk={fmk_type}")
    cmd_str_list.append(f"--modelFile={model_path}")
    cmd_str_list.append(f"--outputFile={model_path}")

    p = subprocess.Popen(cmd_str_list, shell=False)
    status = False
    if p:
        p.wait(timeout=CONVERTER_TIME_OUT)
        ret_value = p.returncode
        status = ret_value >= 0 if ret_value is not None else False
    else:
        logging.error("subprocess popen failed")

    ms_model_path = model_path + ".ms"
    ms_file_exist = os.path.exists(ms_model_path)
    status = status and ms_file_exist

python解析xml :

xml.etree.ElementTree --- ElementTree XML API — Python 3.10.1 文档

两级namespace时findall用第二级

from xml.etree import ElementTree as ET

xml_str = """
    
        
            
                4891
                4892
            
        
    
"""
root = ET.fromstring(xml_str)

p = root.findall(".//{namespace2}loadbalance")

for child in p:
    print(child.tag, child.text)
    for child1 in child:
        print(child1.tag, child1.text)

ip检查标准库

ipaddress — IPv4/IPv6 manipulation library — Python 3.10.1 documentation

代码检查flake8 pylint，自动格式化autopep8

pip安装上述包

flake8/pylint 文件/文件夹名

autopep8 文件名

clear;flake8 --ignore=E1,E501 python/

clear;pylint --disable C0301 python/

autopep8 --in-place --max-line-length 120 setup.py

find ./ -name *.py |xargs -i autopep8 --in-place --max-line-length 120 {}

pylint错误提示代码行上面加注释避免其提示相应错误（示例）：

# pylint: disable=too-many-arguments,too-many-locals

int to bytes

print((255).to_bytes(2, byteorder='little'))

data1 = [1,2,3,4]
arry1 = []
for elem in data1:
    arry1.append((elem).to_bytes(1, byteorder='little'))
data2 =  b''.join(arry1)
print(data2)

Python 字符串前面加u,r,b,f的含义 Python 字符串前面加u,r,b,f的含义 - junwalo - 博客园

pathlib使用

匹配文件夹下文件名

import glob
files = glob.glob('./*.py')

os.mkdir

os.makedirs : mkdir -p

os.path.exists

logging设置 logging - How to log source file name and line number in Python - Stack Overflow 顶层python文件设置一次即可

logging.basicConfig(stream=sys.stdout,
                    format='%(asctime)s,%(msecs)d %(levelname)s [%(filename)s:%(lineno)d] %(message)s',
                    datefmt='%Y-%m-%d:%H:%M:%S',
                    level=logging.DEBUG)

decorator Python 函数装饰器 | 菜鸟教程

读写图像

PIL读写图像

from PIL import Image
import numpy as np

image_file="img_front/id_card_0084_001.jpg"

image = Image.open(image_file)
image_np=np.array(image)

print("image inf: ", image_np.shape, image_np.dtype)

image = Image.fromarray(image_np) # np.uint8, or TypeError: Cannot handle this data type

image.save("./image_save.jpg")

PIL图像在转换为numpy.ndarray后，格式为(h,w,c)，像素顺序为RGB；

OpenCV在cv2.imread()后数据类型为numpy.ndarray，格式为(h,w,c)，像素顺序为BGR。（转自：PyTorch载入图片后ToTensor解读（含PIL和OpenCV读取图片对比） - 木易修 - 博客园）

numpy增加或减少维度

np.expand_dims(x, axis=0)

np.squeeze(x, axis=0)

获取代码绝对路径 （python得到代码所在文件的绝对路径 - cknds - 博客园）

获取 python的include和lib path: How to find out where the Python include directory is? - Stack Overflow

python3 -c 'import sysconfig; print(sysconfig.get_path("include"))'
python3 -c 'import sysconfig; print(sysconfig.get_path("stdlib"))'
TF_INCLUDE_PATH=$(python3 -c 'import tensorflow as tf; print(tf.sysconfig.get_compile_flags()[0].strip("-I"))')
TF_LIB_PATH=$(python3 -c 'import tensorflow as tf; print(tf.sysconfig.get_link_flags()[0].strip("-L"))')

获取当前文件绝对路径

current_path = os.path.abspath(__file__)

current_dir = os.path.dirname(os.path.abspath(__file__))

获取软连接路径：os.path.realpath(path)

os.path.basename(path)获取文件名

python随机数

python: 随机选择 - 筱筱的春天 - 博客园

python strip陷阱

python中strip()，lstrip()，rstrip()函数的讲解_张康的博客-CSDN博客_python中的strip函数

gdb调试

gdb python3 

run xx.py

c

bt

或者

gdb --args  python3.7 xx.py

文件差异比较

import os
import numpy

imgs = os.listdir(r"G:\test\image")
txts = os.listdir(r"G:\test\txt")

# remove postfix .txt or .jpg
imgs_1 = set([x[0:-4] for x in imgs])
txts_1 = set([x[0:-4] for x in txts])

all_name = imgs_1 | txts_1

diff1 = all_name - imgs_1
diff2 = all_name - txts_1

python *args和**kwargs使用np.random.rand需要使用单个数字作为shape，但我们想直接输入一个shape怎么办：

shape = [1,2,3]
dat = np.random.rand(*shape)


def function(a=0,b=1,c=2):
    print(a,b,c)

function(a=1, b=2, c=3)

paras = {"a":5,"b":6,"c":7}
function(**paras)

dat=[1,2,3]
function(*dat)

定义可变参数函数：

python中可变参数个数参数_robin_chenyu的专栏-CSDN博客

example:

def argTest(*args):
    print(args, type(args))
    for i in args:
        print(i)

args=[1,2,3]
argTest(*args)

matplotlib

from matplotlib.pyplot as plt

plt.figure(figsize=(15,13))
plt.plot(dat1[:,0], label='x1')
plt.plot(dat2[:,0], label='x2')
plt.title('Loss')
plt.legend()
plt.show()

逐行读取

with open(file_name) as file:
    for line in file:
        print(line)

文件写入指定换行格式

Python - Replace newline with return carriage/newline in file output - Stack Overflow

默认open打开文件write行+\"\n”实际写入\r\n

For conversion of an existing file to use a specific line termination pattern, you can read the lines while taking advantage of the universal newlines approach. Then you can write the lines using the explicitly specified line terminator for the output file.

from io import open
with open(infname, 'r') as inf, open(outfname, "w+", newline="\r\n") as outf:
    outf.writelines(inf)

包导入相关

Python 3.x | 史上最详解的 导入（import）_神是念着倒-CSDN博客

pyc反汇编

Python .pyc的编译和反编译 - LuckyZLi - 博客园

pip install uncompyle

/usr/local/python3.7/bin/uncompyle6 --help

uncompyle6 models.pyc > models.py

python to c python - Cython setup.py for several .pyx - Stack Overflow

# all .pyx files in a folder
from distutils.core import setup
from Cython.Build import cythonize

setup(
  name = 'MyProject',
  ext_modules = cythonize(["*.pyx"]),
)

python调用shell

Python调用shell命令常用方法 - *陌上花开* - 博客园

import os
os.system('ls -l')

python kill子线程

Python | Different ways to kill a Thread - GeeksforGeeks

Python exit commands: quit(), exit(), sys.exit() and os._exit() - GeeksforGeeks

源码安装

cd Python-3.7.5
mkdir /usr/local/python3.7.5
./configure --prefix=/usr/local/python3.7.5 --enable-loadable-sqlite-extensions --enable-shared
make && make install

异常处理

try:
    data = 1/0
except Exception as err:
    print(str(err))
 

try:
    xxx
except BaseException as e:
    print('catch exception: ', str(e))

TypeError: Object of type xx is not JSON serializable

解决方案：python - How to make a class JSON serializable - Stack Overflow

在使用json.dumps的前提下主要有两个结局方案：

1：不需要改变json.dumps的调用，通过改变类定义，这个适用于dumps为第三方库调用，不受自己控制的情况:在类里面继承dict或者list类，并在__init__里面对其进行初始化。

class FileItem(dict):
    def __init__(self, fname):
        dict.__init__(self, fname=fname)

f = FileItem('tasks.txt')
json.dumps(f)  #No need to change anything here

 也可以继承tuple，但是构造函数需要注意

python - Subclassing tuple with multiple __init__ arguments - Stack Overflow

2，如果dumps是自己调用的，则有很多其他的解决方案了，使用其他json库或者写一个dumper函数例如：

def dumper(obj):
    try:
        return obj.toJSON()
    except:
        return obj.__dict__
print json.dumps(some_big_object, default=dumper, indent=2)

pybind11 C++具有cast能力

  py::array_t& data0 = xx;
  py::array_t data1 = data0;

pip阿里源

[global] 
index-url = http://mirrors.aliyun.com/pypi/simple
[install] 
trusted-host=mirrors.aliyun.com

pip install --no-deps