u012447220
u012447220

Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process

Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process

Mar 20th, 2012

Imagine you’re a budding chef. A data-curious one, of course, so you start by taking a set of foods (pizza, salad, spaghetti, etc.) and ask 10 friends how much of each they ate in the past day.

Your goal: to find natural groups of foodies, so that you can better cater to each cluster’s tastes. For example, your fratboy friends might love wings and beer, your anime friends might love soba and sushi, your hipster friends probably dig tofu, and so on.

So how can you use the data you’ve gathered to discover different kinds of groups?

One way is to use a standard clustering algorithm like k-means or Gaussian mixture modeling (see this previous post for a brief introduction). The problem is that these both assume a fixed number of clusters, which they need to be told to find. There are a couple methods for selecting the number of clusters to learn (e.g., the gap and prediction strength statistics), but the problem is a more fundamental one: most real-world data simply doesn’t have a fixed number of clusters.

That is, suppose we’ve asked 10 of our friends what they ate in the past day, and we want to find groups of eating preferences. There’s really an infinite number of foodie types (carnivore, vegan, snacker, Italian, healthy, fast food, heavy eaters, light eaters, and so on), but with only 10 friends, we simply don’t have enough data to detect them all. (Indeed, we’re limited to 10 clusters!) So whereas k-means starts with the incorrect assumption that there’s a fixed, finite number of clusters that our points come from, no matter if we feed it more data, what we’d really like is a method positing an infinite number of hidden clusters that naturally arise as we ask more friends about their food habits. (For example, with only 2 data points, we might not be able to tell the difference between vegans and vegetarians, but with 200 data points, we probably could.)

Luckily for us, this is precisely the purview of nonparametric Bayes.*

*Nonparametric Bayes refers to a class of techniques that allow some parameters to change with the data. In our case, for example, instead of fixing the number of clusters to be discovered, we allow it to grow as more data comes in.

A Generative Story

Let’s describe a generative model for finding clusters in any set of data. We assume an infinite set of latent groups, where each group is described by some set of parameters. For example, each group could be a Gaussian with a specified mean μi and standard deviation σi , and these group parameters themselves are assumed to come from some base distribution G0 . Data is then generated in the following manner:

  • Select a cluster.
  • Sample from that cluster to generate a new point.

(Note the resemblance to a finite mixture model.)

For example, suppose we ask 10 friends how many calories of pizza, salad, and rice they ate yesterday. Our groups could be:

  • A Gaussian centered at (pizza = 5000, salad = 100, rice = 500) (i.e., a pizza lovers group).
  • A Gaussian centered at (pizza = 100, salad = 3000, rice = 1000) (maybe a vegan group).
  • A Gaussian centered at (pizza = 100, salad = 100, rice = 10000) (definitely Asian).
  • …

When deciding what to eat when she woke up yesterday, Alice could have thought girl, I’m in the mood for pizza and her food consumption yesterday would have been a sample from the pizza Gaussian. Similarly, Bob could have spent the day in Chinatown, thereby sampling from the Asian Gaussian for his day’s meals. And so on.

The big question, then, is: how do we assign each friend to a group?

Assigning Groups

Chinese Restaurant Process

One way to assign friends to groups is to use a Chinese Restaurant Process. This works as follows: Imagine a restaurant where all your friends went to eat yesterday…

  • Initially the restaurant is empty.
  • The first person to enter (Alice) sits down at a table (selects a group). She then orders food for the table (i.e., she selects parameters for the group); everyone else who joins the table will then be limited to eating from the food she ordered.
  • The second person to enter (Bob) sits down at a table. Which table does he sit at? With probability α/(1+α) he sits down at a new table (i.e., selects a new group) and orders food for the table; with probability 1/(1+α) he sits with Alice and eats from the food she’s already ordered (i.e., he’s in the same group as Alice).
  • …
  • The (n+1)-st person sits down at a new table with probability α/(n+α) , and at table k with probability nk/(n+α) , where nk is the number of people currently sitting at table k.

Note a couple things:

  • The more people (data points) there are at a table (cluster), the more likely it is that people (new data points) will join it. In other words, our groups satisfy a rich get richer property.
  • There’s always a small probability that someone joins an entirely new table (i.e., a new group is formed).
  • The probability of a new group depends on α . So we can think of α as a dispersion parameter that affects the dispersion of our datapoints. The lower alpha is, the more tightly clustered our data points; the higher it is, the more clusters we have in any finite set of points.

(Also notice the resemblance between table selection probabilities and a Dirichlet distribution…)

Just to summarize, given n data points, the Chinese Restaurant Process specifies a distribution over partitions (table assignments) of these points. We can also generate parameters for each partition/table from a base distribution G0 (for example, each table could represent a Gaussian whose mean and standard deviation are sampled from G0 ), though to be clear, this is not part of the CRP itself.

Code

Since code makes everything better, here’s some Ruby to simulate a CRP:

Chinese Restaurant Process chinese_restaurant_process.rb 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Generate table assignments for `num_customers` customers, according to # a Chinese Restaurant Process with dispersion parameter `alpha`. # # returns an array of integer table assignments def chinese_restaurant_process(num_customers, alpha)  return [] if num_customers <= 0   table_assignments = [1] # first customer sits at table 1  next_open_table = 2 # index of the next empty table   # Now generate table assignments for the rest of the customers.  1.upto(num_customers - 1) do |i|  if rand < alpha.to_f / (alpha + i)  # Customer sits at new table.  table_assignments << next_open_table  next_open_table += 1  else  # Customer sits at an existing table.  # He chooses which table to sit at by giving equal weight to each  # customer already sitting at a table.   which_table = table_assignments[rand(table_assignments.size)]  table_assignments << which_table  end  end   table_assignments end

And here’s some sample output:

Chinese Restaurant Process chinese_restaurant_process.rb 
1
2
3
4
5
6
7
8
9
10
11
12
13
14

> chinese_restaurant_process(num_customers = 10, alpha = 1) 1, 2, 3, 4, 3, 3, 2, 1, 4, 3 # table assignments from run 1 1, 1, 1, 1, 1, 1, 2, 2, 1, 3 # table assignments from run 2 1, 2, 2, 1, 3, 3, 2, 1, 3, 4 # table assignments from run 3  > chinese_restaurant_process(num_customers = 10, alpha = 3) 1, 2, 1, 1, 3, 1, 2, 3, 4, 5 1, 2, 3, 3, 4, 3, 4, 4, 5, 5 1, 1, 2, 3, 1, 4, 4, 3, 1, 1  > chinese_restaurant_process(num_customers = 10, alpha = 5) 1, 2, 1, 3, 4, 5, 6, 7, 1, 8 1, 2, 3, 3, 4, 5, 6, 5, 6, 7 1, 2, 3, 4, 5, 6, 2, 7, 2, 1

Notice that as we increase α , so too does the number of distinct tables increase.

Polya Urn Model

Another method for assigning friends to groups is to follow the Polya Urn Model. This is basically the same model as the Chinese Restaurant Process, just with a different metaphor.

  • We start with an urn containing αG0(x) balls of “color” x , for each possible value of x . ( G0 is our base distribution, and G0(x) is the probability of sampling x from G0 ). Note that these are possibly fractional balls.
  • At each time step, draw a ball from the urn, note its color, and then drop both the original ball plus a new ball of the same color back into the urn.

Note the connection between this process and the CRP: balls correspond to people (i.e., data points), colors correspond to table assignments (i.e., clusters), alpha is again a dispersion parameter (put differently, a prior), colors satisfy a rich-get-richer property (since colors with many balls are more likely to get drawn), and so on. (Again, there’s also a connection between this urn model and the urn model for the (finite) Dirichlet distribution…)

To be precise, the difference between the CRP and the Polya Urn Model is that the CRP specifies only a distribution over partitions (i.e., table assignments), but doesn’t assign parameters to each group, whereas the Polya Urn Model does both.

Code

Again, here’s some code for simulating a Polya Urn Model:

Polya Urn Model polya_urn_model.rb 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Draw `num_balls` colored balls according to a Polya Urn Model # with a specified base color distribution and dispersion parameter # `alpha`. # # returns an array of ball colors def polya_urn_model(base_color_distribution, num_balls, alpha)  return [] if num_balls <= 0   balls_in_urn = []  0.upto(num_balls - 1) do |i|  if rand < alpha.to_f / (alpha + balls_in_urn.size)  # Draw a new color, put a ball of this color in the urn.  new_color = base_color_distribution.call  balls_in_urn << new_color  else  # Draw a ball from the urn, add another ball of the same color.  ball = balls_in_urn[rand(balls_in_urn.size)]  balls_in_urn << ball  end  end   balls_in_urn end

And here’s some sample output, using a uniform distribution over the unit interval as the color distribution to sample from:

Polya Urn Model polya_urn_model.rb 
1
2
3
4
5
6

> unit_uniform = lambda { (rand * 100).to_i / 100.0 }  > polya_urn_model(unit_uniform, num_balls = 10, alpha = 1) 0.27, 0.89, 0.89, 0.89, 0.73, 0.98, 0.43, 0.98, 0.89, 0.53 # colors in the urn from run 1 0.26, 0.26, 0.46, 0.26, 0.26, 0.26, 0.26, 0.26, 0.26, 0.85 # colors in the urn from run 2 0.96, 0.87, 0.96, 0.87, 0.96, 0.96, 0.87, 0.96, 0.96, 0.96 # colors in the urn from run 3

Code, Take 2

Here’s the same code for a Polya Urn Model, but in R:

Polya Urn Model polya_urn_model.R 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# Return a vector of `num_balls` ball colors according to a Polya Urn Model # with dispersion `alpha`, sampling from a specified base color distribution. polya_urn_model = function(base_color_distribution, num_balls, alpha) {  balls = c()   for (i in 1:num_balls) {  if (runif(1) < alpha / (alpha + length(balls))) {  # Add a new ball color.  new_color = base_color_distribution()  balls = c(balls, new_color)  } else {  # Pick out a ball from the urn, and add back a  # ball of the same color.  ball = balls[sample(1:length(balls), 1)]  balls = c(balls, ball)  }  }   balls }

Here are some sample density plots of the colors in the urn, when using a unit normal as the base color distribution:

Notice that as alpha increases (i.e., we sample more new ball colors from our base; i.e., as we place more weight on our prior), the colors in the urn tend to a unit normal (our base color distribution).

And here are some sample plots of points generated by the urn, for varying values of alpha:

  • Each color in the urn is sampled from a uniform distribution over [0,10]x[0,10] (i.e., a [0, 10] square).
  • Each group is a Gaussian with standard deviation 0.1 and mean equal to its associated color, and these Gaussian groups generate points.

Notice that the points clump together in fewer clusters for low values of alpha, but become more dispersed as alpha increases.

Stick-Breaking Process

Imagine running either the Chinese Restaurant Process or the Polya Urn Model without stop. For each group i , this gives a proportion wi of points that fall into group i .

So instead of running the CRP or Polya Urn model to figure out these proportions, can we simply generate them directly?

This is exactly what the Stick-Breaking Process does:

  • Start with a stick of length one.
  • Generate a random variable β1∼Beta(1,α) . By the definition of the Beta distribution, this will be a real number between 0 and 1, with expected value 1/(1+α) . Break off the stick at β1 ; w1 is then the length of the stick on the left.
  • Now take the stick to the right, and generate β2∼Beta(1,α) . Break off the stick β2 into the stick. Again, w2 is the length of the stick to the left, i.e., w2=(1−β1)β2 .
  • And so on.

Thus, the Stick-Breaking process is simply the CRP or Polya Urn Model from a different point of view. For example, assigning customers to table 1 according to the Chinese Restaurant Process is equivalent to assigning customers to table 1 with probability w1 .

Code

Here’s some R code for simulating a Stick-Breaking process:

Stick-Breaking Process stick_breaking_process.R 
1
2
3
4
5
6
7
8
9
10
11
12
13

# Return a vector of weights drawn from a stick-breaking process # with dispersion `alpha`. # # Recall that the kth weight is # \beta_k = (1 - \beta_1) * (1 - \beta_2) * ... * (1 - \beta_{k-1}) * beta_k # where each $\\beta\_i$ is drawn from a Beta distribution # \beta_i ~ Beta(1, \alpha) stick_breaking_process = function(num_weights, alpha) {  betas = rbeta(num_weights, 1, alpha)  remaining_stick_lengths = c(1, cumprod(1 - betas))[1:num_weights]  weights = remaining_stick_lengths * betas  weights }

And here’s some sample output:

Notice that for low values of alpha, the stick weights are concentrated on the first few weights (meaning our data points are concentrated on a few clusters), while the weights become more evenly dispersed as we increase alpha (meaning we posit more clusters in our data points).

Dirichlet Process

Suppose we run a Polya Urn Model several times, where we sample colors from a base distribution G0 . Each run produces a distribution of colors in the urn (say, 5% blue balls, 3% red balls, 2% pink balls, etc.), and the distribution will be different each time (for example, 5% blue balls in run 1, but 1% blue balls in run 2).

For example, let’s look again at the plots from above, where I generated samples from a Polya Urn Model with the standard unit normal as the base distribution:

Each run of the Polya Urn Model produces a slighly different distribution, though each is “centered” in some fashion around the standard Gaussian I used as base. In other words, the Polya Urn Model gives us a distribution over distributions (we get a distribution of ball colors, and this distribution of colors changes each time) – and so we finally get to the Dirichlet Process.

Formally, given a base distribution G0 and a dispersion parameter α , a sample from the Dirichlet Process DP(G0,α) is a distribution G∼DP(G0,α) . This sample G can be thought of as a distribution of colors in a single simulation of the Polya Urn Model; sampling from G gives us the balls in the urn.

So here’s the connection between the Chinese Restaurant Process, the Polya Urn Model, the Stick-Breaking Process, and the Dirichlet Process:

  • Dirichlet Process: Suppose we want samples xi∼G , where G is a distribution sampled from the Dirichlet Process G∼DP(G0,α) .
  • Polya Urn Model: One way to generate these values xi would be to take a Polya Urn Model with color distribution G0 and dispersion α . ( xi would be the color of the ith ball in the urn.)
  • Chinese Restaurant Process: Another way to generate xi would be to first assign tables to customers according to a Chinese Restaurant Process with dispersion α . Every customer at the nth table would then be given the same value (color) sampled from G0 . ( xi would be the value given to the ith customer; xi can also be thought of as the food at table i , or as the parameters of table i .)
  • Stick-Breaking Process: Finally, we could generate weights wk according to a Stick-Breaking Process with dispersion α . Next, we would give each weight wk a value (or color) vk sampled from G0 . Finally, we would assign xi to value (color) vk with probability wk .

Recap

Let’s summarize what we’ve discussed so far.

We have a bunch of data points pi that we want to cluster, and we’ve described four essentially equivalent generative models that allow us to describe how each cluster and point could have arisen.

In the Chinese Restaurant Process:

  • We generate table assignments g1,…,gn∼CRP(α) according to a Chinese Restaurant Process. ( gi is the table assigned to datapoint i .)
  • We generate table parameters ϕ1,…,ϕn∼G0 according to the base distribution G0 , where ϕk is the parameter for the kth distinct group.
  • Given table assignments and table parameters, we generate each datapoint pi∼F(ϕgi) from a distribution F with the specified table parameters. (For example, F could be a Gaussian, and ϕi could be a parameter vector specifying the mean and standard deviation).

In the Polya Urn Model:

  • We generate colors ϕ1,…,ϕn∼Polya(G0,α) according to a Polya Urn Model. ( ϕi is the color of the ith ball.)
  • Given ball colors, we generate each datapoint pi∼F(ϕi) .

In the Stick-Breaking Process:

  • We generate group probabilities (stick lengths) w1,…,w∞∼Stick(α) according to a Stick-Breaking process.
  • We generate group parameters ϕ1,…,ϕ∞∼G0 from G0 , where ϕk is the parameter for the kth distinct group.
  • We generate group assignments g1,…,gn∼Multinomial(w1,…,w∞) for each datapoint.
  • Given group assignments and group parameters, we generate each datapoint pi∼F(ϕgi) .

In the Dirichlet Process:

  • We generate a distribution G∼DP(G0,α) from a Dirichlet Process with base distribution G0 and dispersion parameter α .
  • We generate group-level parameters xi∼G from G , where xi is the group parameter for the ith datapoint. (Note: this is not the same as ϕi . xi is the parameter associated to the group that the ith datapoint belongs to, whereas ϕk is the parameter of the kth distinct group.)
  • Given group-level parameters xi , we generate each datapoint pi∼F(xi) .

Also, remember that each model naturally allows the number of clusters to grow as more points come in.

Inference in the Dirichlet Process Mixture

So we’ve described a generative model that allows us to calculate the probability of any particular set of group assignments to data points, but we haven’t described how to actually learn a good set of group assignments.

Let’s briefly do this now. Very roughly, the Gibbs sampling approach works as follows:

  • Take the set of data points, and randomly initialize group assignments.
  • Pick a point. Fix the group assignments of all the other points, and assign the chosen point a new group (which can be either an existing cluster or a new cluster) with a CRP-ish probability (as described in the models above) that depends on the group assignments and values of all the other points.
  • We will eventually converge on a good set of group assignments, so repeat the previous step until happy.

For more details, this paper provides a good description. Philip Resnick and Eric Hardisty also have a friendlier, more general description of Gibbs sampling (plus an application to naive Bayes) here.

Fast Food Application: Clustering the McDonald’s Menu

Finally, let’s show an application of the Dirichlet Process Mixture. Unfortunately, I didn’t have a data set of people’s food habits offhand, so instead I took this list of McDonald’s foods and nutrition facts.

After normalizing each item to have an equal number of calories, and representing each item as a vector of (total fat, cholesterol, sodium, dietary fiber, sugars, protein, vitamin A, vitamin C, calcium, iron, calories from fat, satured fat, trans fat, carbohydrates), I ran scikit-learn’s Dirichlet Process Gaussian Mixture Model to cluster McDonald’s menu based on nutritional value.

First, how does the number of clusters inferred by the Dirichlet Process mixture vary as we feed in more (randomly ordered) points?

As expected, the Dirichlet Process model discovers more and more clusters as more and more food items arrive. (And indeed, the number of clusters appears to grow logarithmically, which can in fact be proved.)

How many clusters does the mixture model infer from the entire dataset? Running the Gibbs sampler several times, we find that the number of clusters tends around 11:

Let’s dive into one of these clusterings.

Cluster 1 (Desserts)

Looking at a sample of foods from the first cluster, we find a lot of desserts and dessert-y drinks:

  • Caramel Mocha
  • Frappe Caramel
  • Iced Hazelnut Latte
  • Iced Coffee
  • Strawberry Triple Thick Shake
  • Snack Size McFlurry
  • Hot Caramel Sundae
  • Baked Hot Apple Pie
  • Cinnamon Melts
  • Kiddie Cone
  • Strawberry Sundae

We can also look at the nutritional profile of some foods from this cluster (after z-scaling each nutrition dimension to have mean 0 and standard deviation 1):

We see that foods in this cluster tend to be high in trans fat and low in vitamins, protein, fiber, and sodium.

Cluster 2 (Sauces)

Here’s a sample from the second cluster, which contains a lot of sauces:

  • Hot Mustard Sauce
  • Spicy Buffalo Sauce
  • Newman’s Own Low Fat Balsamic Vinaigrette

And looking at the nutritional profile of points in this cluster, we see that it’s heavy in sodium and fat:

Cluster 3 (Burgers, Crispy Foods, High-Cholesterol)

The third cluster is very burgery:

  • Hamburger
  • Cheeseburger
  • Filet-O-Fish
  • Quarter Pounder with Cheese
  • Premium Grilled Chicken Club Sandwich
  • Ranch Snack Wrap
  • Premium Asian Salad with Crispy Chicken
  • Butter Garlic Croutons
  • Sausage McMuffin
  • Sausage McGriddles

It’s also high in fat and sodium, and low in carbs and sugar

Cluster 4 (Creamy Sauces)

Interestingly, even though we already found a cluster of sauces above, we discover another one as well. These sauces appear to be much more cream-based:

  • Creamy Ranch Sauce
  • Newman’s Own Creamy Caesar Dressing
  • Coffee Cream
  • Iced Coffee with Sugar Free Vanilla Syrup

Nutritionally, these sauces are higher in calories from fat, and much lower in sodium:

Cluster 5 (Salads)

Here’s a salad cluster. A lot of salads also appeared in the third cluster (along with hamburgers and McMuffins), but that’s because those salads also all contained crispy chicken. The salads in this cluster are either crisp-free or have their chicken grilled instead:

  • Premium Southwest Salad with Grilled Chicken
  • Premium Caesar Salad with Grilled Chicken
  • Side Salad
  • Premium Asian Salad without Chicken
  • Premium Bacon Ranch Salad without Chicken

This is reflected in the higher content of iron, vitamin A, and fiber:

Cluster 6 (More Sauces)

Again, we find another cluster of sauces:

  • Ketchup Packet
  • Barbeque Sauce
  • Chipotle Barbeque Sauce

These are still high in sodium, but much lower in fat compared to the other sauce clusters:

Cluster 7 (Fruit and Maple Oatmeal)

Amusingly, fruit and maple oatmeal is in a cluster by itself:

  • Fruit & Maple Oatmeal

Cluster 8 (Sugary Drinks)

We also get a cluster of sugary drinks:

  • Strawberry Banana Smoothie
  • Wild Berry Smoothie
  • Iced Nonfat Vanilla Latte
  • Nonfat Hazelnut
  • Nonfat Vanilla Cappuccino
  • Nonfat Caramel Cappuccino
  • Sweet Tea
  • Frozen Strawberry Lemonade
  • Coca-Cola
  • Minute Maid Orange Juice

In addition to high sugar content, this cluster is also high in carbohydrates and calcium, and low in fat.

Cluster 9 (Breakfast Foods)

Here’s a cluster of high-cholesterol breakfast foods:

  • Sausage McMuffin with Egg
  • Sausage Burrito
  • Egg McMuffin
  • Bacon, Egg & Chees Biscuit
  • McSkillet Burrito with Sausage
  • Big Breakfast with Hotcakes

Cluster 10 (Coffee Drinks)

We find a group of coffee drinks next:

  • Nonfat Cappuccino
  • Nonfat Latte
  • Nonfat Latte with Sugar Free Vanilla Syrup
  • Iced Nonfat Latte

These are much higher in calcium and protein, and lower in sugar, than the other drink cluster above:

Cluster 11 (Apples)

Here’s a cluster of apples:

  • Apple Dippers with Low Fat Caramel Dip
  • Apple Slices

Vitamin C, check.

And finally, here’s an overview of all the clusters at once (using a different clustering run):

No More!

I’ll end with a couple notes:

  • Kevin Knight has a hilarious introduction to Bayesian inference that describes some applications of nonparametric Bayesian techniques to computational linguistics (though I don’t think he ever quite says “nonparametric Bayes” directly).
  • In the Chinese Restaurant Process, each customer sits at a single table. The Indian Buffet Process is an extension that allows customers to sample food from multiple tables (i.e., belong to multiple clusters).
  • The Chinese Restaurant Process, the Polya Urn Model, and the Stick-Breaking Process are all sequential models for generating groups: to figure out table parameters in the CRP, for example, you wait for customer 1 to come in, then customer 2, then customer 3, and so on. The equivalent Dirichlet Process, on the other hand, is a parallel model for generating groups: just sample G∼DP(G0,alpha) , and then all your group parameters can be independently generated by sampling from G at once. This duality is an instance of a more general phenomenon known as de Finetti’s theorem.

And that’s it.

Posted by Edwin Chen Mar 20th, 2012


原文地址：http://blog.echen.me/2012/03/20/infinite-mixture-models-with-nonparametric-bayes-and-the-dirichlet-process/

你可能感兴趣的:(Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process)

  • WebMagic：强大的Java爬虫框架解析与实战 Aaron_945 Javajava爬虫开发语言
    文章目录引言官网链接WebMagic原理概述基础使用1.添加依赖2.编写PageProcessor高级使用1.自定义Pipeline2.分布式抓取优点结论引言在大数据时代，网络爬虫作为数据收集的重要工具，扮演着不可或缺的角色。Java作为一门广泛使用的编程语言，在爬虫开发领域也有其独特的优势。WebMagic是一个开源的Java爬虫框架，它提供了简单灵活的API，支持多线程、分布式抓取，以及丰富的
  • python结束子进程_如何清除python中的子进程 weixin_39995943 python结束子进程
    我们使用python进程来管理长时间运行的python子进程。有时需要终止子进程。kill命令不会完全终止进程，只会使其失效。运行以下脚本将演示此行为。importsubprocessp=subprocess.Popen(['sleep','400'],stdout=subprocess.PIPE,shell=False)或者p=subprocess.Popen('sleep400',stdout
  • python获取子进程返回值_Python对进程Multiprocessing子进程返回值 weixin_39752157 python获取子进程返回值
    在实际使用多进程的时候，可能需要获取到子进程运行的返回值。如果只是用来存储，则可以将返回值保存到一个数据结构中；如果需要判断此返回值，从而决定是否继续执行所有子进程，则会相对比较复杂。另外在Multiprocessing中，可以利用Process与Pool创建子进程，这两种用法在获取子进程返回值上的写法上也不相同。这篇中，我们直接上代码，分析多进程中获取子进程返回值的不同用法，以及优缺点。初级用法
  • AI大模型的架构演进与最新发展 季风泯灭的季节 AI大模型应用技术二人工智能架构
    随着深度学习的发展，AI大模型（LargeLanguageModels,LLMs）在自然语言处理、计算机视觉等领域取得了革命性的进展。本文将详细探讨AI大模型的架构演进，包括从Transformer的提出到GPT、BERT、T5等模型的历史演变，并探讨这些模型的技术细节及其在现代人工智能中的核心作用。一、基础模型介绍：Transformer的核心原理Transformer架构的背景在Transfo
  • spring security中几大组件的作用和执行顺序 阿信在这里 javaspring
    springsecurity中几大组件的作用和执行顺序在SpringSecurity中，AuthenticationProvider、GroupPermissionEvaluator、PermissionEvaluator、AbstractAuthenticationProcessingFilter、DefaultMethodSecurityExpressionHandler和ManageSecu
  • Scanpy源码浅析之pp.normalize_total 何物昂
    版本导入Scanpy,其版本为'1.9.1'，如果你看到的源码和下文有差异，其可能是由于版本差异。importscanpyasscsc.__version__#'1.9.1'例子函数pp.normalize_total用于Normalizecountspercell，其源代码在scanpy/preprocessing/_normalization.py我们通过一个简单例子来了解该函数主要功能:将一
  • golang学习笔记--MPG模型 xxzed golang#学习笔记学习笔记golang
    MPG模式：M（Machine）：操作系统的主线程P（Processor）：协程执行需要的资源（上下文context），可以看作一个局部的调度器，使go代码在一个线程上跑，他是实现从N：1到N：M映射的关键G（Goroutine）：协程，有自己的栈。包含指令指针（instructionpointer）和其它信息（正在等待的channel等等），用于调度。一个P下面可以有多个G1、当前程序有三个M,
  • 基于Python执行lua脚本 xu-jssy Python自动化脚本pythonlua自动化rpa
    一、依赖安装pipinstalllupa二、源码将lua文件存放在base_path路径，将lua文件名称（不包含后缀名）传递给lua_runner函数即可importmultiprocessingimportlupa#lua文件存放位置base_path='D:\\test\\lua'classLuaFuncion:#创建Lua运行时环境lua=lupa.LuaRuntime(unpack_re
  • 影刀RPA与WPS文档协同办公：实现高效自动化处理的策略与实践 enter回车键 影刀RPA
    摘要随着数字化转型的深入，企业对于办公自动化的需求日益增长。影刀RPA（RoboticProcessAutomation）与WPS文档的协同办公提供了一种高效、自动化的解决方案。本文旨在探讨影刀RPA与WPS文档如何配合使用，以实现工作流程的自动化，提高办公效率，并为企业带来实际效益。引言影刀RPA作为一种自动化工具，能够模拟人类用户的行为，执行重复性高、规则性强的工作任务。而WPS文档作为办公软
  • python+adb 0o一人情 adb命令Python项目python开发语言
    #!/usr/bin/pythonenv#-*-coding:utf-8-*-importosimportsysimportsubprocessfromtimeimportsleepimportlogginglogging.basicConfig(level=logging.DEBUG)classScreenCapture():defget_screen_size(self):"""获取手机分辨率
  • 2.8.5Django --8.2 单表操作 寒暄_HX
    Django目录：https://www.jianshu.com/p/dc36f62b3dc5Yuan先生-Django模型层（1）Django与SQLAlchemy的ORM操作本质上是一样的，但是语法略有不同，如果是用Django进行开发最好使用原生的ORM或者直接使用原生SQL。创建表app06创建模型在app06中的models.py文件内，新建一个模板。one_exa.app06.mode
  • ResNet的半监督和半弱监督模型 Valar_Morghulis
    Billion-scalesemi-supervisedlearningforimageclassificationhttps://arxiv.org/pdf/1905.00546.pdfhttps://github.com/facebookresearch/semi-supervised-ImageNet1K-models/权重在timm中也有：https://hub.fastgit.org/r
  • MySQL数据库全面学习之（上篇） 一心只为学 数据库mysql学习
    Windows服务--启动MySQLnetstartmysql--创建Windows服务sccreatemysqlbinPath=mysqld_bin_path(注意：等号与值之间有空格)连接与断开服务器mysql-h地址-P端口-u用户名-p密码SHOWPROCESSLIST--显示哪些线程正在运行SHOWVARIABLES--显示系统变量信息数据库操作--查看当前数据库SELECTDATABA
  • 什么是 PHP? 为什么用 PHP? 谁在用 PHP? m0_37438181 永远学习php开发语言
    一、什么是PHP？PHP（HypertextPreprocessor，超文本预处理器）是一种广泛应用于Web开发的通用开源脚本语言。PHP主要用于服务器端编程，可以嵌入HTML中，与数据库进行交互，生成动态网页内容。它具有以下特点：简单易学：语法相对简单，容易上手，对于初学者来说是一个不错的选择。跨平台性：可以在多种操作系统上运行，如Windows、Linux、Unix等。丰富的函数库：提供了大量
  • ModuleNotFoundError: No module named ‘timm.layers‘ 忽略不计， BUGpythonYOLO目标检测人工智能深度学习
    解决方式：把fromtimm.layersimportDropPath这个修改为fromtimm.models.layersimportDropPath即可。
  • 座舱交互的下一个时代 高工智能汽车 交互物联网人工智能
    为了满足座舱信息娱乐的更高性能要求，几乎所有的一线品牌都在准备“换芯”。去年开始，不少车型开始推动传统的分布式座舱仪表和中控电子架构进入域控制器时代，高通成为大赢家。今年6月，特斯拉也正式官宣，即将推出的新款ModelS将配备能够运行PS5游戏机性能的AMD芯片，包括专门定制的AMDRyzenCPU和独立的Navi23图形处理器。最新消息，特斯拉将率先在中国市场生产的ModelY高性能版车型换装A
  • 【笔记】扩散模型（七）：Latent Diffusion Models（Stable Diffusion）论文解读与代码实现 LittleNyima DiffusionModels笔记stablediffusionAIGC人工智能
    论文链接：High-ResolutionImageSynthesiswithLatentDiffusionModels官方实现：CompVis/latent-diffusion、CompVis/stable-diffusion这一篇文章的内容是LatentDiffusionModels（LDM），也就是大名鼎鼎的StableDiffusion。先前的扩散模型一直面临的比较大的问题是采样空间太大，学
  • vue vueUse利用useInfiniteScroll API 实现虚拟滚动 小白小白从不日白 Vue3vue.js前端
    前言中文网地址：开始使用|VueUse中文网官网地址：VueUse元素的无限滚动。useInfiniteScroll详细解析地址效果组件封装
  • Python 课程8-多线程编程和多进程编程 可愛小吉 Python教學python开发语言threadingmultiprocessing
    前言在现代编程中，处理并发任务是提高程序性能的关键之一。Python提供了多线程（threading）和多进程（multiprocessing）两种方式来实现并发编程。多线程适用于I/O密集型任务，而多进程则更适合CPU密集型任务。通过这两种技术，你可以高效地处理大规模数据、加速程序执行并优化资源利用。在本篇详细教程中，我们将讨论如何使用Python的threading模块实现多线程，以及如何使用
  • 通过进程Id终止进程 好学松鼠 C++进程C++Windows编程
    #include#include//通过进程ID终止进程BOOLTerminateProcessFromID(DWORDdwID){BOOLbRet=FALSE;//打开进程HANDLEhProcess=::OpenProcess(PROCESS_ALL_ACCESS,FALSE,dwID);if(hProcess!=NULL){//终止进程bRet=::TerminateProcess(hPro
  • Webpack插件核心原理 gogo2027 webpack
    引言围绕Webpack打包流程中最核心的机制就是所谓的Plugin机制。所谓插件即是webpack生态中最关键的部分，它为社区用户提供了一种强有力的方式来直接触及webpack的编译过程(compilationprocess)。今天，我们来聊聊Webpack中必不可少的核心Plugin机制～Plugin本质上在Webpack编译阶段会为各个编译对象初始化不同的Hook，开发者可以在自己编写的Plu
  • 多模态大模型微调Qwen-VL微调及日志 Messi^ 人工智能-大模型应用python人工智能深度学习
    %pipinstallmodelscope-U%pipinstalltransformersacceleratetiktoken-U%pipinstalleinopstransformers_stream_generator-U%pipinstallpillow-U%pipinstalltorchvision%pipinstallmatplotlib-Ufrommodelscopeimport(s
  • 思维导图-ProcessOn 佛系猿
    今天介绍一款特别好用的流程图、思维导图软件ProcessOn用途：在线画流程图、思维导图、UI原型图、UML、网络拓扑图、组织结构图等各种模板供你选择image支持团队协作支持不同格式下载image更多查看官网最后附上做的效果图：image
  • 查看 CPU架构类型 BYAPESS windows
    打开cmd窗口—>输入echo%PROCESSOR_ARCHITECTURE%接口显示，本人的是AMD64
  • 基于XTDrone的ZD550+Mid360实现 夜雨拾年 无人机
    前言本文是对ZD550搭载Mid360激光雷达Gazebo仿真平台搭建记录的实现文件导入先下载提供的文件链接:https://pan.baidu.com/s/1reqGCcQOj1T_tGBY3EZWpw?pwd=328c提取码:328c将文件夹ZD550_Mid360中的5个文件夹都添加到PX4_Firmware/Tools/sitl_gazebo/models目录下，其中models目录下可能
  • 大规模语言模型从理论到实践 vLLM推理框架实践 AGI通用人工智能之禅 计算科学神经计算深度学习神经网络大数据人工智能大型语言模型AIAGILLMJavaPython架构设计AgentRPA
    大规模语言模型从理论到实践：vLLM推理框架实践1.背景介绍1.1问题的由来随着大规模语言模型（LargeLanguageModels,vLLMs）的发展，从简单的语言生成到复杂的多模态任务，这些模型的能力得到了显著提升。然而，如何高效地利用这些模型进行推理成为了新的挑战。传统的方法往往受限于模型的输入长度、计算资源的限制以及缺乏有效的任务分解策略。为了解决这些问题，vLLM推理框架应运而生，旨在
  • appium中遇到WebDriverException: Message: An unknown server-side error occurred while processing the ... Kingtester
    selenium.common.exceptions.WebDriverException:Message:Anunknownserver-sideerroroccurredwhileprocessingthecommand.Originalerror:Anewsessioncouldnotbecreated.Details:sessionnotcreated:pleaseclose'com.te
  • python sanic orm_sanic中使用tortoise-orm Mr浪子相依 pythonsanicorm
    #models.pyfromtortoise.modelsimportModelfromtortoiseimportfieldsclassUser(Model):id=fields.IntField(pk=True,,source_field="userID")name=fields.CharField(max_length=100)date_field=fields.DateTimeField(
  • 一天认识一个硬件之CPU 哲伦贼稳妥 一天认识一个硬件IT技术电脑硬件电脑运维硬件工程其他
    CPU，全称为中央处理器（CentralProcessingUnit），是计算机硬件系统的核心部件之一，负责执行计算机程序中的指令和处理数据。它相当于计算机的大脑，今天就来给大家分享一下台式机和笔记本大脑的对比。性能差异核心数量和频率：台式机CPU通常支持更多的核心数量和更高的运行频率，这使得它们在处理多线程任务和多任务处理方面更具优势。性能释放：笔记本CPU受限于散热和供电条件，功耗通常较低，导
  • vue中给打包的文件指定自定义文件名以及加上哈希值解决每次打包上线存在缓存问题 miao_zz vuevue
    vue中给打包的文件指定自定义文件名以及加上哈希值解决每次打包上线存在缓存问题vue.config.jsvue.config.jsconstport=process.env.port||8081//端口constTimestamp=newDate().getTime();constMiniCssExtractPlugin=require("mini-css-extract-plugin")modu
  • Linux的Initrd机制 被触发 linux
    Linux 的 initrd 技术是一个非常普遍使用的机制，linux2.6 内核的 initrd 的文件格式由原来的文件系统镜像文件转变成了 cpio 格式，变化不仅反映在文件格式上， linux 内核对这两种格式的 initrd 的处理有着截然的不同。本文首先介绍了什么是 initrd 技术，然后分别介绍了 Linux2.4 内核和 2.6 内核的 initrd 的处理流程。最后通过对 Lin
  • maven本地仓库路径修改 bitcarter maven
    默认maven本地仓库路径：C:\Users\Administrator\.m2 修改maven本地仓库路径方法：     1.打开E:\maven\apache-maven-2.2.1\conf\settings.xml     2.找到        
  • XSD和XML中的命名空间 darrenzhu xmlxsdschemanamespace命名空间
    http://www.360doc.com/content/12/0418/10/9437165_204585479.shtml http://blog.csdn.net/wanghuan203/article/details/9203621 http://blog.csdn.net/wanghuan203/article/details/9204337 http://www.cn
  • Java 求素数运算 周凡杨 java算法素数
    网络上对求素数之解数不胜数，我在此总结归纳一下，同时对一些编码，加以改进，效率有成倍热提高。 第一种：   原理: 6N(+-)1法         任何一个自然数，总可以表示成为如下的形式之一： 6N，6N+1，6N+2，6N+3，6N+4，6N+5 (N=0，1，2，…)   
  • java 单例模式 g21121 java
    想必单例模式大家都不会陌生，有如下两种方式来实现单例模式：   class Singleton { private static Singleton instance=new Singleton(); private Singleton(){} static Singleton getInstance() { return instance; }
  • Linux下Mysql源码安装 510888780 mysql
    1.假设已经有mysql-5.6.23-linux-glibc2.5-x86_64.tar.gz (1)创建mysql的安装目录及数据库存放目录       解压缩下载的源码包，目录结构，特殊指定的目录除外：           
  • 32位和64位操作系统 墙头上一根草 32位和64位操作系统
    32位和64位操作系统是指：CPU一次处理数据的能力是32位还是64位。现在市场上的CPU一般都是64位的，但是这些CPU并不是真正意义上的64 位CPU，里面依然保留了大部分32位的技术，只是进行了部分64位的改进。32位和64位的区别还涉及了内存的寻址方面，32位系统的最大寻址空间是2 的32次方= 4294967296（bit）= 4（GB）左右，而64位系统的最大寻址空间的寻址空间则达到了
  • 我的spring学习笔记10-轻量级_Spring框架 aijuans Spring 3
    一、问题提问：     → 请简单介绍一下什么是轻量级？     轻量级（Leightweight）是相对于一些重量级的容器来说的，比如Spring的核心是一个轻量级的容器，Spring的核心包在文件容量上只有不到1M大小，使用Spring核心包所需要的资源也是很少的，您甚至可以在小型设备中使用Spring。  
  • mongodb 环境搭建及简单CURD antlove WebInstallcurdNoSQLmongo
    一 搭建mongodb环境 1. 在mongo官网下载mongodb 2. 在本地创建目录 "D:\Program Files\mongodb-win32-i386-2.6.4\data\db" 3. 运行mongodb服务 [mongod.exe --dbpath "D:\Program Files\mongodb-win32-i386-2.6.4\data\
  • 数据字典和动态视图 百合不是茶 oracle数据字典动态视图系统和对象权限
    数据字典（data dictionary）是 Oracle 数据库的一个重要组成部分，这是一组用于记录数据库信息的只读（read-only）表。随着数据库的启动而启动,数据库关闭时数据字典也关闭   数据字典中包含   数据库中所有方案对象（schema object）的定义(包括表，视图，索引，簇，同义词，序列，过程，函数，包，触发器等等) 数据库为一
  • 多线程编程一般规则 bijian1013 javathread多线程java多线程
           如果两个工两个以上的线程都修改一个对象，那么把执行修改的方法定义为被同步的，如果对象更新影响到只读方法，那么只读方法也要定义成同步的。        不要滥用同步。如果在一个对象内的不同的方法访问的不是同一个数据，就不要将方法设置为synchronized的。
  • 将文件或目录拷贝到另一个Linux系统的命令scp bijian1013 linuxunixscp
    一.功能说明        scp就是security copy，用于将文件或者目录从一个Linux系统拷贝到另一个Linux系统下。scp传输数据用的是SSH协议，保证了数据传输的安全，其格式如下：        scp 远程用户名@IP地址：文件的绝对路径
  • 【持久化框架MyBatis3五】MyBatis3一对多关联查询 bit1129 Mybatis3
    以教员和课程为例介绍一对多关联关系，在这里认为一个教员可以叫多门课程，而一门课程只有1个教员教，这种关系在实际中不太常见，通过教员和课程是多对多的关系。   示例数据：   地址表：   CREATE TABLE ADDRESSES ( ADDR_ID INT(11) NOT NULL AUTO_INCREMENT, STREET VAR
  • cookie状态判断引发的查找问题 bitcarter formcgi
    先说一下我们的业务背景： 1.前台将图片和文本通过form表单提交到后台，图片我们都做了base64的编码，并且前台图片进行了压缩 2.form中action是一个cgi服务 3.后台cgi服务同时供PC，H5，APP 4.后台cgi中调用公共的cookie状态判断方法（公共的，大家都用，几年了没有问题） 问题：（折腾两天。。。。） 1.PC端cgi服务正常调用，cookie判断没
  • 通过Nginx,Tomcat访问日志(access log)记录请求耗时 ronin47
    一、Nginx通过$upstream_response_time $request_time统计请求和后台服务响应时间 nginx.conf使用配置方式： log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_r
  • java-67- n个骰子的点数。 把n个骰子扔在地上，所有骰子朝上一面的点数之和为S。输入n，打印出S的所有可能的值出现的概率。 bylijinnan java
    public class ProbabilityOfDice { /** * Q67 n个骰子的点数 * 把n个骰子扔在地上，所有骰子朝上一面的点数之和为S。输入n，打印出S的所有可能的值出现的概率。 * 在以下求解过程中，我们把骰子看作是有序的。 * 例如当n=2时，我们认为（1，2）和（2，1）是两种不同的情况 */ private stati
  • 看别人的博客，觉得心情很好 Cb123456 博客心情
       以为写博客，就是总结，就和日记一样吧，同时也在督促自己。今天看了好长时间博客:    职业规划:    http://www.iteye.com/blogs/subjects/zhiyeguihua      android学习:    1.http://byandby.i
  • [JWFD开源工作流]尝试用原生代码引擎实现循环反馈拓扑分析 comsci 工作流
        我们已经不满足于仅仅跳跃一次，通过对引擎的升级，今天我测试了一下循环反馈模式，大概跑了200圈，引擎报一个溢出错误      在一个流程图的结束节点中嵌入一段方程，每次引擎运行到这个节点的时候，通过实时编译器GM模块，计算这个方程，计算结果与预设值进行比较，符合条件则跳跃到开始节点，继续新一轮拓扑分析，直到遇到
  • JS常用的事件及方法 cwqcwqmax9 js
    事件 描述 onactivate 当对象设置为活动元素时触发。 onafterupdate 当成功更新数据源对象中的关联对象后在数据绑定对象上触发。 onbeforeactivate 对象要被设置为当前元素前立即触发。 onbeforecut 当选中区从文档中删除之前在源对象触发。 onbeforedeactivate 在 activeElement 从当前对象变为父文档其它对象之前立即
  • 正则表达式验证日期格式 dashuaifu 正则表达式IT其它java其它
    正则表达式验证日期格式 function isDate(d){ var v = d.match(/^(\d{4})-(\d{1,2})-(\d{1,2})$/i); if(!v) { this.focus(); return false; } } <input value="2000-8-8" onblu
  • Yii CModel.rules() 方法 、validate预定义完整列表、以及说说验证 dcj3sjt126com yii
    public array rules () {return} array 要调用 validate() 时应用的有效性规则。 返回属性的有效性规则。声明验证规则，应重写此方法。 每个规则是数组具有以下结构：array('attribute list', 'validator name', 'on'=>'scenario name', ...validation
  • UITextAttributeTextColor = deprecated in iOS 7.0 dcj3sjt126com ios
    In this lesson we used the key "UITextAttributeTextColor" to change the color of the UINavigationBar appearance to white. This prompts a warning "first deprecated in iOS 7.0." Ins
  • 判断一个数是质数的几种方法 EmmaZhao Mathpython
    质数也叫素数，是只能被1和它本身整除的正整数，最小的质数是2，目前发现的最大的质数是p=2^57885161-1【注1】。 判断一个数是质数的最简单的方法如下： def isPrime1(n): for i in range(2, n): if n % i == 0: return False return True 但是在上面的方法中有一些冗余的计算，所以
  • SpringSecurity工作原理小解读 坏我一锅粥 SpringSecurity
      SecurityContextPersistenceFilter   ConcurrentSessionFilter   WebAsyncManagerIntegrationFilter   HeaderWriterFilter   CsrfFilter   LogoutFilter   Use
  • JS实现自适应宽度的Tag切换 ini JavaScripthtmlWebcsshtml5
    效果体验：http://hovertree.com/texiao/js/3.htm   该效果使用纯JavaScript代码，实现TAB页切换效果，TAB标签根据内容自适应宽度，点击TAB标签切换内容页。 HTML文件代码： <!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml"
  • Hbase Rest API : 数据查询 kane_xie RESThbase
    hbase（hadoop）是用java编写的，有些语言（例如python）能够对它提供良好的支持，但也有很多语言使用起来并不是那么方便，比如c#只能通过thrift访问。Rest就能很好的解决这个问题。Hbase的org.apache.hadoop.hbase.rest包提供了rest接口，它内嵌了jetty作为servlet容器。   启动命令：./bin/hbase rest s
  • JQuery实现鼠标拖动元素移动位置（源码+注释） 明子健 jqueryjs源码拖动鼠标
    欢迎讨论指正！   print.html代码： <!DOCTYPE html> <html> <head> <meta http-equiv=Content-Type content="text/html;charset=utf-8"> <title>发票打印</title> &l
  • Postgresql 连表更新字段语法 update qifeifei PostgreSQL
    下面这段sql本来目的是想更新条件下的数据，可是这段sql却更新了整个表的数据。sql如下： UPDATE tops_visa.visa_order SET op_audit_abort_pass_date = now() FROM tops_visa.visa_order as t1 INNER JOIN tops_visa.visa_visitor as t2 ON t1.
  • 将redis,memcache结合使用的方案? tcrct rediscache
    公司架构上使用了阿里云的服务，由于阿里的kvstore收费相当高，打算自建，自建后就需要自己维护，所以就有了一个想法，针对kvstore(redis)及ocs(memcache)的特点，想自己开发一个cache层，将需要用到list，set，map等redis方法的继续使用redis来完成，将整条记录放在memcache下，即findbyid，save等时就memcache，其它就对应使用redi
  • 开发中遇到的诡异的bug wudixiaotie bug
    今天我们服务器组遇到个问题： 我们的服务是从Kafka里面取出数据，然后把offset存储到ssdb中，每个topic和partition都对应ssdb中不同的key，服务启动之后，每次kafka数据更新我们这边收到消息，然后存储之后就发现ssdb的值偶尔是-2,这就奇怪了，最开始我们是在代码中打印存储的日志，发现没什么问题，后来去查看ssdb的日志，才发现里面每次set的时候都会对同一个key
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他