librelist archives

« back to archive

"pip wheel" subcommand

"pip wheel" subcommand

From:
Marcus Smith
Date:
2012-10-08 @ 16:29
Daniel and others:
I'll be attempting the "pip wheel" sub-command this week.
https://gist.github.com/3828683
I'll let you know how it goes.
Marcus

Re: [wheel] "pip wheel" subcommand

From:
Paul Moore
Date:
2012-10-10 @ 14:35
On 8 October 2012 17:29, Marcus Smith <qwcode@gmail.com> wrote:
> Daniel and others:
> I'll be attempting the "pip wheel" sub-command this week.
> https://gist.github.com/3828683
> I'll let you know how it goes.

I've just added a couple of comments based on some work I've been
doing recently building wheels using the current --wheel-cache
approach.

Paul.

Re: [wheel] "pip wheel" subcommand

From:
Sebastien Douche
Date:
2012-10-10 @ 17:11
On Wed, Oct 10, 2012 at 4:35 PM, Paul Moore <p.f.moore@gmail.com> wrote:
> On 8 October 2012 17:29, Marcus Smith <qwcode@gmail.com> wrote:
>> Daniel and others:
>> I'll be attempting the "pip wheel" sub-command this week.
>> https://gist.github.com/3828683
>> I'll let you know how it goes.
>
> I've just added a couple of comments based on some work I've been
> doing recently building wheels using the current --wheel-cache
> approach.

Hi there,
Can we do use more than one index? It's a terrible limitation (since 4
years, I must use Zope KGS[1] to by-pass that).



[1] KGS create an index *and* a find-links from a list of packages, so
I can use 2 index (PyPI and a private).


-- 
Sebastien Douche <sdouche@gmail.com>
Twitter: @sdouche / G+: +sdouche

Re: [wheel] "pip wheel" subcommand

From:
Daniel Holth
Date:
2012-10-10 @ 17:24
On Wed, Oct 10, 2012 at 1:11 PM, Sebastien Douche <sdouche@gmail.com> wrote:
> On Wed, Oct 10, 2012 at 4:35 PM, Paul Moore <p.f.moore@gmail.com> wrote:
>> On 8 October 2012 17:29, Marcus Smith <qwcode@gmail.com> wrote:
>>> Daniel and others:
>>> I'll be attempting the "pip wheel" sub-command this week.
>>> https://gist.github.com/3828683
>>> I'll let you know how it goes.
>>
>> I've just added a couple of comments based on some work I've been
>> doing recently building wheels using the current --wheel-cache
>> approach.
>
> Hi there,
> Can we do use more than one index? It's a terrible limitation (since 4
> years, I must use Zope KGS[1] to by-pass that).
>
> [1] KGS create an index *and* a find-links from a list of packages, so
> I can use 2 index (PyPI and a private).

That is not quite the problem domain for wheel, which is just a binary
package format like egg.

Wheel can be used make pip a little more like buildout: first, you
collect built versions of just the packages you want into folders of
.whl files, then quickly install from those pre-built packages without
consulting an index.

Re: [wheel] "pip wheel" subcommand

From:
Marcus Smith
Date:
2012-10-10 @ 15:55
I responded.  is there a way to get notifications on gists?  I'm not seeing
it?  :  (

On Wed, Oct 10, 2012 at 7:35 AM, Paul Moore <p.f.moore@gmail.com> wrote:

> On 8 October 2012 17:29, Marcus Smith <qwcode@gmail.com> wrote:
> > Daniel and others:
> > I'll be attempting the "pip wheel" sub-command this week.
> > https://gist.github.com/3828683
> > I'll let you know how it goes.
>
> I've just added a couple of comments based on some work I've been
> doing recently building wheels using the current --wheel-cache
> approach.
>
> Paul.
>

Re: [wheel] "pip wheel" subcommand

From:
Michele Lacchia
Date:
2012-10-10 @ 16:06
Unfortunately there is not. I just starred it but, for example, I was not
seeing this update...

2012/10/10 Marcus Smith <qwcode@gmail.com>

> I responded.  is there a way to get notifications on gists?  I'm not
> seeing it?  :  (
>
>
> On Wed, Oct 10, 2012 at 7:35 AM, Paul Moore <p.f.moore@gmail.com> wrote:
>
>> On 8 October 2012 17:29, Marcus Smith <qwcode@gmail.com> wrote:
>> > Daniel and others:
>> > I'll be attempting the "pip wheel" sub-command this week.
>> > https://gist.github.com/3828683
>> > I'll let you know how it goes.
>>
>> I've just added a couple of comments based on some work I've been
>> doing recently building wheels using the current --wheel-cache
>> approach.
>>
>> Paul.
>>
>
>


-- 
Michele Lacchia

Re: [wheel] "pip wheel" subcommand

From:
Marcus Smith
Date:
2012-10-10 @ 16:27
Daniel:

I didn't follow this comment: " pip could include the "run any setup.py
under setuptools" as a script.

Marcus

Re: "pip wheel" subcommand

From:
Marcus Smith
Date:
2012-10-15 @ 18:52
I have a working version of "pip wheel". need to add tests and tweak.

one core question though... about "--force-rebuild"
at first, I was thinking to use pep425.get_supported to determine whether I
had something already in my ./wheel dir,  but that's not really accurate.
so the next idea is to use  bdist_wheel.get_archive_basename (which
involves instantiating a distutils command)

thoughts?

Marcus

On Mon, Oct 8, 2012 at 9:29 AM, Marcus Smith <qwcode@gmail.com> wrote:

> Daniel and others:
> I'll be attempting the "pip wheel" sub-command this week.
> https://gist.github.com/3828683
> I'll let you know how it goes.
> Marcus
>

Re: "pip wheel" subcommand

From:
Marcus Smith
Date:
2012-10-15 @ 18:59
at first, I was thinking to use pep425.get_supported to determine whether I
> had something already in my ./wheel dir,  but that's not really accurate.
>

to be clear, what I meant was that I would construct all the possibilities
of wheel names using help from get_supported, and not rebuild, if anything
was there, but that's not what we want.

Re: [wheel] Re: "pip wheel" subcommand

From:
Daniel Holth
Date:
2012-10-15 @ 19:14
On Mon, Oct 15, 2012 at 2:59 PM, Marcus Smith <qwcode@gmail.com> wrote:
>
>
>
>> at first, I was thinking to use pep425.get_supported to determine whether
>> I had something already in my ./wheel dir,  but that's not really accurate.
>
>
> to be clear, what I meant was that I would construct all the possibilities
> of wheel names using help from get_supported, and not rebuild, if anything
> was there, but that's not what we want.

The rebuild feature is more like whether you add the wheel dir to
--find-links or whether you allow wheels at all for (a package | all
packages) when you are doing the build.

"pip wheel --target x y"

- builds wheels for y into x
 - for y and each dependency of y that is not installed

If you don't include the target dir x in --find-links then pip will
naturally rebuild everything that isn't installed into the active
environment, every time.

If you do include x, then pip will find the existing wheel and unpack
it into build/ as part of the normal dependency resolution process.

Then there's --ignore-installed which I assume you've kept.

Did you refactor into a "packagefindingcommand" that Install and Wheel
inherit from? I noticed there is also a remove_option() for the parser
so you could in fact implement by inheriting from InstallCommand
without having to expose all its arguments.

Daniel

Re: [wheel] Re: "pip wheel" subcommand

From:
Marcus Smith
Date:
2012-10-15 @ 19:49
"pip wheel" is just a build tool. it always ignores installations right now
(--ignore-installed is always true in the RequirementSet I'm building)

--force-rebuild is about whether you will overwrite or just keep what's in
your wheels directory ('./wheels' by default if you don't specify)
(you can point your wheel destination at your find-links source (that
you'll use later during installs) if you want.)
so this option needs to know in advance what the built filename will be,
hence my question.



> Did you refactor into a "packagefindingcommand" that Install and Wheel
> inherit from?


I thought about it, but no.  for now, the only cross-command sharing is a
new options module that constructs all the options using
optparse.make_option.
then the various commands can put together the options they want w/o
reconstructing them in a redundant way.

Marcus

Re: [wheel] Re: "pip wheel" subcommand

From:
Marcus Smith
Date:
2012-10-15 @ 20:04
the intended workflow:
"pip wheel -r requirements.txt --wheel-dir /my/wheel/stash"
"pip install --use-wheel --no-index -r requirements
--find-links=/my/wheel/stash

On Mon, Oct 15, 2012 at 12:49 PM, Marcus Smith <qwcode@gmail.com> wrote:

>
> "pip wheel" is just a build tool. it always ignores installations right
> now (--ignore-installed is always true in the RequirementSet I'm building)
>
> --force-rebuild is about whether you will overwrite or just keep what's in
> your wheels directory ('./wheels' by default if you don't specify)
> (you can point your wheel destination at your find-links source (that
> you'll use later during installs) if you want.)
> so this option needs to know in advance what the built filename will be,
> hence my question.
>
>
>
>> Did you refactor into a "packagefindingcommand" that Install and Wheel
>> inherit from?
>
>
> I thought about it, but no.  for now, the only cross-command sharing is a
> new options module that constructs all the options using
> optparse.make_option.
> then the various commands can put together the options they want w/o
> reconstructing them in a redundant way.
>
> Marcus
>

Re: [wheel] Re: "pip wheel" subcommand

From:
Daniel Holth
Date:
2012-10-15 @ 20:07
On Mon, Oct 15, 2012 at 3:49 PM, Marcus Smith <qwcode@gmail.com> wrote:
>
> "pip wheel" is just a build tool. it always ignores installations right now
> (--ignore-installed is always true in the RequirementSet I'm building)
>
> --force-rebuild is about whether you will overwrite or just keep what's in
> your wheels directory ('./wheels' by default if you don't specify)
> (you can point your wheel destination at your find-links source (that you'll
> use later during installs) if you want.)
> so this option needs to know in advance what the built filename will be,
> hence my question.

I think it works. If the wheel command has ./wheels (consider:
wheelbase, wheelhouse) in find-links during wheel building, then it
will not rebuild if it can find a compatible wheel. If ./wheels is not
in find-links, then it will always rebuild, overwriting anything with
the same filename.

Another option would be to ignore all wheels even if they are in
find-links for the purpose of wheel rebuilding, or to ignore remote
wheels.

You got the intended workflow right, except that in the future you
would use wheel by default, and you might want to specify "use wheel
when possible", "use wheel if no sdist", and "never use wheel". You
could even give sdists an index in the wheel ranking system and adjust
(putting sdists in the front, the back, or the middle of the list) as
desired.

Re: [wheel] Re: "pip wheel" subcommand

From:
Marcus Smith
Date:
2012-10-15 @ 20:22
> If ./wheels is not
> in find-links, then it will always rebuild, overwriting anything with
> the same filename.
>

I'm not sure we're on the same page here.   you're use of "it will"
confuses me.  we're discussing what "will be" the case for "pip wheel".
in order for "pip wheel" to not rebuild the wheel file, I have to *add*
something new to "pip wheel", that's not there currently.
as it is, "pip wheel" will always just rebuild.
to do otherwise, I have to check my "--wheel-dir" to see if it contains
what I'm going to build.
in order to do that, I have to know the filename for what I'm going to
build, hence the question about using bdist_wheel.get_archive_basename



> You got the intended workflow right, except that in the future you
> would use wheel by default,
>

right, down the road.

Re: [wheel] Re: "pip wheel" subcommand

From:
Daniel Holth
Date:
2012-10-15 @ 20:41
On Mon, Oct 15, 2012 at 4:22 PM, Marcus Smith <qwcode@gmail.com> wrote:
>
>>
>> If ./wheels is not
>> in find-links, then it will always rebuild, overwriting anything with
>> the same filename.
>
>
> I'm not sure we're on the same page here.   you're use of "it will" confuses
> me.  we're discussing what "will be" the case for "pip wheel".
> in order for "pip wheel" to not rebuild the wheel file, I have to *add*
> something new to "pip wheel", that's not there currently.
> as it is, "pip wheel" will always just rebuild.
> to do otherwise, I have to check my "--wheel-dir" to see if it contains what
> I'm going to build.
> in order to do that, I have to know the filename for what I'm going to
> build, hence the question about using bdist_wheel.get_archive_basename

Does "pip wheel" honor --use-wheel and --find-links, and can it
"build" a wheel from a wheel?

The way "pip install --wheel-cache target depname" works is that any
dep that is not installed or already available as a wheel, is built as
a wheel and then dumped into target.

You do not need to know the filename of the wheel you are rebuilding.
You are only rebuilding "pyramid" and it will output as whatever its
default wheel filename is.

If you used --use-wheel --find-links on an index that already contains
a wheel for pyramid, it won't be rebuilt.

So in the current "install --use-wheel" patch assuming pypi has no
wheels and nothing is installed,

"Don't rebuild"

pip install --use-wheel --find-links wheelbase --wheel-cache wheelbase pyramid

"Rebuild"

pip install --use-wheel --wheel-cache wheelbase pyramid


In other words the only way to decide whether you need to build a
wheel is to run the PackageFinder. When it finds an sdist, you need to
build a wheel. When it finds a wheel, you don't. By disabling wheel in
the packagefinder or using an index that contains no wheels (omitting
your local wheelbase from --find-links), you automatically rebuild all
the wheels.

Re: [wheel] Re: "pip wheel" subcommand

From:
Paul Moore
Date:
2012-10-15 @ 21:34
On 15 October 2012 21:22, Marcus Smith <qwcode@gmail.com> wrote:
>> If ./wheels is not in find-links, then it will always rebuild, overwriting
>> anything with the same filename.
>
> I'm not sure we're on the same page here.   you're use of "it will" confuses
> me.  we're discussing what "will be" the case for "pip wheel". in order for
> "pip wheel" to not rebuild the wheel file, I have to *add* something new to
> "pip wheel", that's not there currently.as it is, "pip wheel" will always just
> rebuild. to do otherwise, I have to check my "--wheel-dir" to see if it contains
> what I'm going to build. in order to do that, I have to know the filename for
> what I'm going to build, hence the question about using
> bdist_wheel.get_archive_basename

If I understand him correctly, Daniel is still looking at this from
the perspective of wheels as a (mostly transparent) cache of the
results of building from sdist. Whereas I (and you, I think) am
looking at it from the perspective of wheels being a binary
distribution format you build and later use.

For my workflow, "pip wheel -r req.txt" is exactly the command I'd use
to build wheels. I don't want pip wheel to decide not to rebuild a
wheel after I've asked it to, so the code as it stands is fine. Having
it skip building something that is already present (the change you
asked about at the start of this thread) is a possibly-useful
optimisation to save an unnecessary build, but it's hardly essential.
And it'd need to be optional, there would certainly be times I'd want
to force a rebuild (a corrupt file, for example).

Daniel's suggestion is pushing the design back towards the "pip
install --wheel-cache" approach where wheels are build "on demand" as
part of an install, and saved to speed up later installs. I can see
why this would be the best approach in certain workflows, but I don't
like it, personally. What Daniel seems to be after is for pip wheel to
take a --find-links parameter like pip install, so that it can see if
it *needs* to build a wheel to satisfy an install request, and only
build if it does. So, we would have

(1) pip wheel --find-links /my/local/repository --wheel-dir
/my/local/repository -r req.txt
(2) pip wheel --find-links /my/local/repository --use-wheel -r req.txt

Step (1) ensures the cache /my/local/repository has all the necessary
wheels present so that step (2) will not build anything from source.

The added complexity, in terms of extra code in pip, as well as in
terms of the command structure (why does pip wheel need a --find-links
argument, which is an installation parameter?) doesn't seem worth it
to me.

Paul.

Re: [wheel] Re: "pip wheel" subcommand

From:
Daniel Holth
Date:
2012-10-15 @ 22:35
On Oct 15, 2012, at 5:34 PM, Paul Moore <p.f.moore@gmail.com> wrote:

> On 15 October 2012 21:22, Marcus Smith <qwcode@gmail.com> wrote:
>>> If ./wheels is not in find-links, then it will always rebuild, overwriting
>>> anything with the same filename.
>> 
>> I'm not sure we're on the same page here.   you're use of "it will" confuses
>> me.  we're discussing what "will be" the case for "pip wheel". in order for
>> "pip wheel" to not rebuild the wheel file, I have to *add* something new to
>> "pip wheel", that's not there currently.as it is, "pip wheel" will always just
>> rebuild. to do otherwise, I have to check my "--wheel-dir" to see if it
contains
>> what I'm going to build. in order to do that, I have to know the filename for
>> what I'm going to build, hence the question about using
>> bdist_wheel.get_archive_basename
> 
> If I understand him correctly, Daniel is still looking at this from
> the perspective of wheels as a (mostly transparent) cache of the
> results of building from sdist. Whereas I (and you, I think) am
> looking at it from the perspective of wheels being a binary
> distribution format you build and later use.
> 
> For my workflow, "pip wheel -r req.txt" is exactly the command I'd use
> to build wheels. I don't want pip wheel to decide not to rebuild a
> wheel after I've asked it to, so the code as it stands is fine. Having
> it skip building something that is already present (the change you
> asked about at the start of this thread) is a possibly-useful
> optimisation to save an unnecessary build, but it's hardly essential.
> And it'd need to be optional, there would certainly be times I'd want
> to force a rebuild (a corrupt file, for example).
> 
> Daniel's suggestion is pushing the design back towards the "pip
> install --wheel-cache" approach where wheels are build "on demand" as
> part of an install, and saved to speed up later installs. I can see
> why this would be the best approach in certain workflows, but I don't
> like it, personally. What Daniel seems to be after is for pip wheel to
> take a --find-links parameter like pip install, so that it can see if
> it *needs* to build a wheel to satisfy an install request, and only
> build if it does. So, we would have
> 
> (1) pip wheel --find-links /my/local/repository --wheel-dir
> /my/local/repository -r req.txt
> (2) pip wheel --find-links /my/local/repository --use-wheel -r req.txt
> 
> Step (1) ensures the cache /my/local/repository has all the necessary
> wheels present so that step (2) will not build anything from source.
> 
> The added complexity, in terms of extra code in pip, as well as in
> terms of the command structure (why does pip wheel need a --find-links
> argument, which is an installation parameter?) doesn't seem worth it
> to me.

Find-links is a package finder parameter.

The goal is to have everything packaged locally and to later do the actual
install offline or on another machine without consulting the 'net. It's 
much faster for latency reasons as well. It is two phase, not transparent.

Iiuc we will see the code soon and can have a concrete discussion.