RangeFrom, Part 1..: History and background

Introduction

While working on a Clippy lint I began to think that something was wrong about the implementations of the RangeFrom type. To get my thought out somewhere I thought to write a post about it. Since my thoughts have mainly been about one of the design points decided early on I decided it was a good idea to first dive into the history of the type, to learn how it what bacame what we use today and how it will evolve with the new range types.

In part two I will then yap a bit about my thoughts about the design and give some of my opinions on it, although from the things I focus on in this post it might be quite clear what I want to yap about.

What is a range?

A range is a mathematical construct sometimes also called a interval, in this post we will only integer ranges. There are few different kinds of ranges which exist in rust.

  • .. A full range, this is equivalent to [,+], in Rust this type is called RangeFull.
  • ..m A upwards bounded range, this is equivalent to [,𝑚), in Rust this type is called RangeTo.
  • n.. A upwards unbounded range, this is equivalent to [𝑛,+], in Rust this type is called RangeFrom.
  • n..m A bounded range, this is equivalent to [𝑛,𝑚), in Rust this type is called Range
  • There are also inclusive versions: ..=m equivalent to [,𝑚] and n..=m equivalent to [𝑛,𝑚].

These can be used for various things, they were originally added for slicing, for example you can get a slice pointing to field 2, 3 and 4 with x[2..5].

Some of them can also to iterate over so if you want to print 1 through 10 you can write:

for i in 1..=10 {
  println!("{i}");
}

It is only RangeFrom, Range and its inclusive versions that implement iterator (either directly or in the future indirectly).

For the rest of this post series I will mainly focus on the RangeFrom type.

The history of the RangeFrom type in Rust.

Original Implementation

Rust got ranges for slicing in RFC0198 which was implemented in rust#17318. This was followed up with RFC0439 started on which made quite a few changes to the cmp and ops modules. This RFC changes many of the traits in those modules to the traits that we use today. And tucked away in a small chapter is the changes that are proposed to add ranges as actual type and not just a special case for slicing: Slice revisions.

The implementation of this RFC is tracked in rust#19148 and on contributor @nrc writes that he will implement ranges and indexing using ranges. There is a bit of discussion in the issue for adding range notation rust#19794 about if the range types should implement Iterator or have a Self::iter method that returns a iterator which we might get back to later.

This leads to the pull request that adds the new range types rust#19858. Here is the full implementation of RangeFrom in core from that PR[^1]:

/// A range which is only bounded below.
#[deriving(Copy)]
#[lang="range_from"]
pub struct RangeFrom<Idx> {
    /// The lower bound of the range (inclusive).
    pub start: Idx,
}

impl<Idx: Clone + Step> Iterator<Idx> for RangeFrom<Idx> {
    #[inline]
    fn next(&mut self) -> Option<Idx> {
        // Deliberately overflow so we loop forever.
        let result = self.start.clone();
        self.start.step();
        return Some(result);
    }
}

Here we can see that from the very first implementation we had deliberate overflow in case we get to the end of the index type. Interestingly the Step trait was also introduced in this pull request and as of writing is still unstable.

But why was this decision to make the iterator overflow made? Its a bit hard to answer as a lot of the discussion in those time happened on IRC. While there are archives of the logs such as rust-irc-logs, the logs from the period around the time of the implementation of ranges is missing.

I reached out to @nrc to hear if he could remember any of the reasoning even though it was years ago. He remembered there being discussion about whether n.. should mean that it loops forever or n..=MAX and they decided on the former. As for reasons his best bet was that it matches a C-style for loop.

Another thing with this implementation is if overflow checks are on it will never yield the last value in integer. If self.start is equal 255 it will panic before yielding 255. This leads to some oddities that result in issues such as rust#25708.

Discussions about the overflow

The above pull request was merged on , two days later on an issue was opened about it: "Take care of boundaries in Step trait" #20249 by @bluss.

In the issue thread the following two messages jumps out to me:

This was intentional and discussed with a bunch of people on irc. The majority felt that an open ended interval should repeat forever, rather than being bounded by the maximum. …

@nrc Comment

But the intervals won’t repeat this way – not with the coming overflow rules and overflow checks. It seems wiser to insert the boundary check manually (and I think to terminate iteration there).

@bluss Comment

The discussion in the thread seems to land on that in the end not overflowing will be the responsibility of the end user, and then closed after they made overflow cause a panic in the default debug profile.

Later in 2015 another issue about the overflow is opened: rust#25696 this and the before mentioned rust#25708 is later closed by a pull request that adds a note about the overflow: rust#32592 The pull request was made because of a question asked on the /r/rust subreddit: I don't understand why this for loop is giving me an "arithmetic operation overflowed". In the thread on Reddit explained by some helpful users, and there are some discussions about if it is an oversight, but this is waved away by a now deleted user saying it was discussed a lot, that discussion is not one I have been able to find.

In he pull request @alexchricton as the reviewer asks:

This will only create an endless loop for the range 0.., right? Something like 1.. will overflow to 0 at some point which should stop iteration?

@alexchricton Comment

Which the implementor tbu- responds to with

No, it’s always an endless loop (if it doesn’t panic), …

@tbu- Comment

In the end the added note is quite non-commitial and reads as

/// Note: Currently, no overflow checking is done for the iterator
/// implementation; if you use an integer range and the integer overflows, it
/// might panic in debug mode or create an endless loop in release mode. This
/// overflow behavior might change in the future.

A few years later on a pull request ammending the note is posted: rust#72368. The pull requests changes the note and stabilises it such that it cannot be changed going forward. The new note reads as follows:

/// *Note*: Overflow in the [`Iterator`] implementation (when the contained
/// data type reaches its numerical limit) is allowed to panic, wrap, or
/// saturate. This behavior is defined by the implementation of the [`Step`]
/// trait. For primitive integers, this follows the normal rules, and respects
/// the overflow checks profile (panic in debug, wrap in release). Note also
/// that overflow happens earlier than you might assume: the overflow happens
/// in the call to `next` that yields the maximum value, as the range must be
/// set to a state to yield the next value.

This is a documentation about how it works and makes no changes to the behaviour of the actual iterator.

And this is also what the documentation for std::ops::RangeFrom is today.

In November 2023 a API Change Proposal (ACP) got opened addressing the overflow behaviour of RangeFrom: libs-team#304. As of writing this no consensus about changes have been reached though there have been some discussions.

RFC3550: new_range

In November 2023 a pre-RFC is opened with the title “Fixing Range by 2027” in this pre-RFC there is a proposal to fix some issues with the range types. The two main issues is that range types are not Copy and that the iterator is directly implemented on the Range* types. (I said that we would come back to it) Most of the comment in the thread are about those two issues. There is one comment about the overflowing behaviour which argues that it probably should not overflow and end at T::MAX, but no one picks it up and gives it more discussion. The pull request for the RFC is opened on , and then merged on and is added to the RFC book as 3550-new-range. The RFC leaves the overflowing behaviour of RangeFrom as a unresolved question for the libs-api team. The libs-api team takes the question up for the meeting on , the minutes are available on HackMD. There is a bit confusion that leads to the question being taken up twice seemingly, but in the end it seems like they decide to mostly keep the current implementation, except they decided that the final value should also be emitted with overflow checks turned on, and with them off it should just overflow. @Amanieu writes a summation on the issue: summary. Then with the release of Rust 1.96.0 on the new types are stabilised.

Conclusion of Part 1..

This is where we are as of writing this post.

If you think I have missed any important parts of the history of RangeFrom please let me know and I will try update this post or have it in my followup post.

Thanks for reading, I will probably write part 2 soon. Any comments can as always be directed to the email listed in my about page.

I extend my aplogies to everyone who have had to listen to me yap about this, especially maya who helped with feedback on the post.

Footnotes

  1. We might get back to that #[deriving(Copy)] as well